[Flex-help] Issues with yy_scan_string() and unput()

[William B. <wil...@gm...>]

Hi flex folks,

I'm working on an interpreter that allows macro substitution. The
language allows macros to be written between backtick and single
quote, and replaced inline with their expansions. (Like C's #define.)
So if the macro "foo" is defined to contain the text "bar", writing
sum(`foo') should be expanded to sum(bar). The macros are supposed do
pure text replacement, without any semantics.

They can also be nested, and a token should be able to consist
partially of text originally in the source code and partially of text
from a macro expansion. I've implemented this behavior with a start
condition stack and a parallel stack of expanded text. While within a
set of nested macros, I keep track of all the state without using
flex's facilities, but once we get back to the INITIAL state I need to
put the final expanded text back into the input stream for reading
under the usual rules.

I considered using a buffer stack or at least one level of additional
buffer, but then tokens wouldn't be able to cross buffer boundaries.
Instead, I'm trying to unput() the expanded text back into yytext.
yytext is declared as an array of the default YYLMAX size rather than
a pointer.

This actually works fine when the input is from stdin. When input
comes in via yy_scan_string, I get a "flex scanner push-back overflow"
error after a few characters. Curiously, while the example below
doesn't show it, the number of characters seems to depend on how much
text is in the string before the macro appears.

The documentation as I read it says that this shouldn't happen, and
I'm wondering whether it's because I'm compiling the scanner as C++. I
can't easily port my stack-handling code to C, so it's hard to check.
Is this a known issue with C++ compilation? Or am I misreading the
docs?

Any information or assistance you can offer is very welcome.

Thanks
Will

=========================
Details and reproducible example:
I'm using flex 2.5.35 and g++ 4.8.4 (the Ubuntu version, 4.8.4-2ubuntu1~14.04).

My actions are C++ code, but I'm compiling the C scanner with g++
rather than use the C++ class interface.

I've attached a minimal example that reproduces this behavior. When
it's compiled and run, I get this:
    wbrannon@ip-10-0-0-87:~$ flex -o lex.yy.cpp ado.fl
    wbrannon@ip-10-0-0-87:~$ g++ -std=c++11 -g -O3 -fPIC -Wall
-pedantic -o test lex.yy.cpp
    wbrannon@ip-10-0-0-87:~$ ./test
    17
    16
    15
    14
    13
    flex scanner push-back overflow
    wbrannon@ip-10-0-0-87:~$