After looking at the problems with YYLMAX several users reported (YYLMAX is reached and falls over). I realized that the problem is with the -l option passed to flex. With the -l option, flex forces yytext to be the size of YYLMAX, which due to a flex mishap is always set to the size of 8192 bytes (this is against the AT&T standard, that allows YYLMAX to be changed).
Without the -l option, flex utilizes a dynamic size buffer for yytext, so incredibly large lines can be parsed. This can also be set with the %pointer directive to flex (which unfortunately is overridden by the -l option).
So we have three options:
1) Remove support for AT&T lex
2) Change flex to function correctly with -l concerning YYLMAX
3) Ignore the problem and list the bug as a feature
1) We can optimize for flex usage only and allows us a dynamic yytext
2) Flex becomes a little more AT&T compiliant with -l
3) No work
1) Will lose AT&T support which seems to give us much better performance for those platforms that do have AT&T lex
2) Will require folks to use a patched flex or more recent version of flex
3) Problem exists
My two cents:
1) is extremely attractive, since the -l option to flex is mentioned to reduce performance. Also flex is available for all platforms, whereas lex is not. Our largest user base is Linux which doesn't use lex.
2) I can't believe that this problem has been hit before so something is wrong here. Also requiring a patched flex may cause problems for builders
3) is extremely attractive, since generally this problem occurs for data that really doesn't need to be parsed anyway (e.g. huge data files, binaries, etc..)
Open to discussion, comments anyone?