openbor Tools

Remove all recursive calls to pp_parser_parse() and introduce persistent subparsers.

Now the preprocessor can really return tokens one at a time!

Fixed detection of normal vs. function-style #defines. Before, directives like this were being misdetected as functions:
#define NUM (16*1024)

Fixed the line numbers reported by pp_lexer.

Fixed a buffer overflow in pp_lexer for tokens longer than 512 characters.

Also, this is revision 3000!!!!! Hooray!

Fixed the parsing of function-style macros with parameters that are not a single identifier, such as 'func(32)' and 'func(foo+bar)'.

Treat backslash-escaped line breaks as whitespace in pp_lexer. This improves the preprocessor's conformance to the C standard in corner cases.

MAJOR refactoring of the preprocessor (pp_parser). Tokens are now emitted one at a time by pp_parser_emit_token(). pp_parser_parse() just calls that function until EOF.

It's worth noting that they're not quite emitted one at a time, because including and macro expansion still call pp_parser_parse() on the subparser. But this still moves the preprocessor a step closer to eliminating the token buffer.

Replaced the emit() calls in pp_parser_readline() with use of the overread property.

Now pp_parser_parse() is the only preprocessor function that uses emit() directly.

r2994 caused the actual contents of all macros to be stored incorrectly. Fixed.

Fixed the preprocessor bug mentioned in the commit message for r2989, in which the line under a #define with only a name (e.g. '#define FOO') would be interpreted as part of the #define and not processed otherwise.