[Flex-devel] Flex code updates

[Joe K. <kr...@ni...>]

My original intent was better option and code organization, which could 
not easily be handled as small patches. In the process, I developed an 
understanding of the Flex code and made changes that are worth 
contributing before trying to make significant changes to the overall 
skeleton and code generation process.

Here is a list of changes I made that may be useful for the current Flex 
code. It would be easier to avoid splitting each change into a separate 
patch. Maye I can start by grouping the most favorable small changes 
into a single patch. A few changes were mainly aesthetic, like renaming 
the "guts" structure. Some are useful, but not so important, like adding 
run-time NLS support.

I hope that line-wrapping doesn't make this too ugly.
Joe
------------------------------------------------------------------------

  * New macro: YY_ISATTY(). This allows you to define your own method to 
check
    for an interactive session, because isatty() is very non-portable.

  * NLS support for messages in generated code, in the text domain
    "flex-runtime". This is how the newer Bison code works.

  * Updated generation of main(): It now handles external tables, and 
supports
    all scanner types, and the optional use of C++ <iostream> for I/O.

  * Added tests for successful malloc/realloc where it was absent. A new
    function was added to report allocation errors by function name, to 
avoid
    the need for multiple NLS-translation strings.

  * Renamed the reentrant state object was renamed from yyguts_t to 
yyobject_t.
    All of the struct members no longer have the yy prefix, because it is
    not needed when encapsulating them in a struct. (Ideally, the C++ and
    yyobject_t names should all match)

  * YY_DECL is never defined by the flex code. Instead, the header has a
    conditional prototype, and the function body has a conditional 
declaration.
    This makes the main lex functions easier to read, and avoids adding
    "#undef YY_DECL" to the header file. Something like this:

    header:
    #ifndef YY_DECL
    int yylex(...);
    #endif

    function source:
    #ifdef YY_DECL
    YYDECL
    #else
    int yylex(...)
    #endif
    {
    ...

  * A new #define YY_ISATTY() macro was added, because 'isatty()' is the 
least
    portable function, but the user's code may be able to provide 
another means
    of checking for interactive input. The macro could instead be name
    "YY_IS_INTERACTIVE()".

  * An option to use C++ stream I/O was added. This allows C++ users to 
use a
    standard lexer with C++ I/O, and it allows use of the C++ class with 
standard
    C I/O.

  * Converted K&R syntax to ANSI in all source code.

  * Removed K&R option for the generated code. For now, it still has the 
options
    keywords, but reports that it is obsolete. This also avoids the need for
    the yyconst macros.

Bugs Fixed:

  * Flex was mis-detecting the use of REJECT and yywrap by matching 
sub-strings
    instead of whole words. (Bug added when m4-quote escaping was 
implemented.)

  * [1544933] input() and unput() now work before calling yylex(). An init
    function was created to hold some of the set-up code at the beginning of
    yylex, which is called by the input() and unput() handlers if needed.
    The newer POSIX spec are a bit unclear, but seem to indicate that 
calling
    input() before yylex() is valid.

  * [2043583] Avoid signed/unsigned comparisons.

  * [1990170] Check the return value of write() in ECHO and issue error on
    failures, similar to input().

  * [2125513] Replaced direct calls to yy_fatal_error() with the macro 
version.

  * [2203641] Added `default:' to all switch statements (for purists).

  * [2178663] Added "#include <cstdio>" for the C++ scanner.

  * [1783536] Fixed yywrap definition to include an argument only for the
    reentrant scanner. (This is really just to avoid warnings, and allow for
    broken compilers that don't like an empty macro argument value.)

  * [2040664] Use YY_CURRENT_BUFFER_LVALUE instead of YY_CURRENT_BUFFER in
    cases where NULL makes no sense. (i.e. left of `->'). This avoids 
warinings
    about possible use of NULL pointers with strict arg checking.

Internal design changes:

  * The 2D yynxt_tbl was converted to 1D, for better uniformity. This was
    already the case for the serialized version of the 2D array, so it 
actually
    simplifies number of the code-generation conditionals. However, the
    resulting 2D table code is a bit easier to read, but it's not meant to
    read.

  * New function yylex_init_state() initializes I/O buffers, and allows 
input()
    and unput() to be used before the first call to yylex().

  * The internal reentrant struct is no called `yyobject_t' instead of
    `yyguts_t'. The skeleton no longer uses the YY_G() macro. Instead, m4
    processing converts the code, and avoids the need for declaring the
    yyobject pointer by expressing a global as:

        (((struct yyobject_t*)yyscanner)->global)

  * The internal scan.l is now more careful at handling the 'no' prefix.