[Flex-devel] Various issues with flex source code and functionality

[Joe K. <kr...@ni...>]

There are several issues in the flex source code that should be 
addressed, but which don't have an obvious answer.

The stdinit option is supposed to initialize I/O to stdin and stdout, 
although those may not be compile-time constants on some compilers. The 
strange thing is that yylex() starts by setting yyin,yyout to 
stdin,stdout if they are NULL. The only thing that stdinit could do is 
allow for using input() to get stdin data before calling yylex, but that 
doesn't work in flex anyhow (bug #618177). So, this option seems useless 
unless the input and unput functions are updated to work outside of yylex().

On a related issue, flex uses yyinput() instead of input() for C++ 
scanners, to avoid a name conflict with the 'input' stream. However, 
there is no such stream name. Maybe it was from an early implementation 
of C++. Besides, all C++ names are protected by namespace prefixes. 
Also, input() is a macro in traditional lex, although POSIX says it 
cannot be redefined. (Does anyone have access to the full POSIX specs?) 
My idea is to make the actual function name yyinput(), so it has the 
yy-prefix namespace, and make input() a macro. C++ code that uses 
yyinput() will still work, and input() will then be a macro that can be 
renamed or even skipped.

According to documentation, the lex-compat option is supposed to make 
unput() non-redefinable, but this is not actually any change to unput() 
in the generated code. I think the idea of POSIX disallowing 
redefinition of unput, input, and output is mainly because there is no 
specification on how to redefine them. So, I still think using macros as 
with the original lex is a good idea, but that the docs should warn that 
changing them is non-portable. We could also add the output() feature to 
be more POSIX compatible.

The docs say that yylineno is only supported with the lex-compat option, 
but this seems no longer the case. But, yylineno and yycolumn both need 
some work to make them fully functional.

Flex has gettext NLS macros everywhere except the generated code. It 
would be good to implement it there, bit it means adding an NLS option 
to flex itself.

Joe Krahn