stream input

[Giuseppe Attardi]

I need to create a lexer reading data from an ifstream:

istream is(argv[1]);
quex::tokenizer_it qlex(&is, "UTF-8");

This worked in version 0.64.8, using

quex --iconv -b 4\
            --bet wchar_t \
            --token-policy single \
            --token-memory-management-by-user

In version 0.65.4 instead I get this error:

Error: Quex engine constructor has received a standard input handle!
Error: Standard input has to be treated like buffered input. Please,
Error: consult the documentation or the example 'stdinlexer.cpp' in
Error: directory $QUEX_PATH/demo/010 (> make stdinlexer.exe). Please,
Error: review the other (non-stdin) examples in this directory to
Error: conclude how to read encoded data via stdin.

Note also that the example in question is number 009 and not 010.
Fix also the English: "to conclude" -> "for learning"

[Giuseppe Attardi]

You suggested to look at example 009 for a solution.

However, what the example does is to read data and to push it to the lexer.
That does not work for me. I want the lexer to pull data from the stream, not viceversa, since I am building a data-driven pipeline.
Each stage pulls data from the previous one.

Here is a simple main function that does what I need, and works in 0.64.8

int main(int argc, char** argv)
{
// (*) create token
quex::Token token;
// (*) create the lexical analyser
istream* is = argc > 1 ? new ifstream(argv[1], ios::in) : &cin;
quex::tokenizer_it qlex(is, "UTF-8");

// (*) loop until the 'termination' token arrives
qlex.token_p_switch(&token);
while (true) {
    // (*) get next token from the token stream
    const QUEX_TYPE_TOKEN_ID TokenID = qlex.receive();

    // (*) check against 'termination'
    if (TokenID == QUEX_TKN_TERMINATION)
      break;
    else if (TokenID == QUEX_TKN_EOS) {
      cout << endl;
    } else {
      // (*) print out token
      cout << quex::unicode_to_char(token.text) << endl;
    }
}
}