Re: [Flex-help] Python Lexical Analysis

[Philip H. <her...@go...>]

On 3 August 2010 07:16, Marcel Laverdet <ma...@la...> wrote:
>
>
> Can't you have a global stack in your scanner to keep track of the
> indentation levels? Something like:
>
> [ \t]+ {
>  size_t top_len = yyextra->indent_stack.top();
>  if (top_len < yyleng) {
>    yyextra->indent_stack.push(yyleng);
>    return INDENT;
>  } else if (top_len > yyleng) {
>    yyextra->indent_stack.pop();
>    return DEDENT;
>  }
> }
>

Ah that makes more sense i just was having problems seeing how it
could look within flex and i thought it could be too messy even for a
first pass basic setup.

> You will need to add proper start conditions to only let this fire after
> you return a NEWLINE. Additionally you need to handle cases like this:
>
>  foo
>      bar
>        etc
>    var

Yeah handling the case after etc could be tricky but I'll play with
this now and let you guys know.

>
> I think after scanning "etc" it should return DEDENT, DEDENT, INDENT; but
> the sample lex example above won't handle that correctly. You should be
> able to pull this off with clever start conditions, or YY_USER_ACTION
> however.

I have never needed to use any big flex trickery like YY_USER_ACTION
before but i will read up on the manual for it now.

Some people have expressed interest in the lexer and parser code
already and its available over at
http://code.redbrain.co.uk/cgit.cgi/gcc-dev/ in the Python branch. I
will however extract out the parser code and post it to comp.compilers
and host it on my blog for people to do whatever they want with it
when its ready.

--Phil