Re: [Flex-help] Building a push parser

[Jerome H. <jer...@gm...>]

Whoa, sweet! One of the authors of the book I use as a reference answered
me!

Thank you for your answers! Would you mind going a bit deeper?

For the "push-lexer" use case, I am thinking about the microcontroller
world, where I don't have threads (although it IS a 32bit MCU). Or do you
recommend other solutions (i.e. not lex&yacc) for online embedded protocol
parsing?
Right now I am scanning for end of line and I push complete lines with
yy_scan_bytes, but it feels silly.
I'll give a try to implementing a "come back later, I might have new data"
function. Do you reckon it is something that might be merged?


For the second question does "the state is in the grammar, not the
tokenizer" mean that the tokenizer does not tokenize any more and passes
only string that are analyzed in the parser or does it work with start
states? And in the case of start states, is there a recommended way of
doing it?

Cheers,
Jérôme.


On Sat, Jan 23, 2021 at 7:59 PM John Levine <jo...@ta...> wrote:

> In article <
> CAE...@ma...> you
> write:
> >I would like to build a push parser, and would like the lexer to behave in
> >accordance.
>
> While that is a reasonable thing to do, flex doesn't do that. When I
> wrote flex&bison in 2009 I noted that bison can create push parsers
> but flex can't.  As far as I can see, nothing has changed there.
>
> While it would be possible in principle to modify flex to do push scanning,
> it'd be a lot of work.  Every environment supports threads these days so
> I would fake it by running the scanner and parser in its own thread.
>
> >Also, on another point, I sometimes feel that I need to enter a new start
> >state but this is not recognizable by the lexer, but only by the parser.
> Am
> >I doing something wrong? Is the grammar I am trying to implement broken?
> Or
> >is it legitimate?
>
> That is quite common. There's lots of situations where the information
> about the state is in the grammar, not the tokenizer.
>
> R's,
> John
>