Re: [Flex-help] How to scan for token with no predefined value?
flex is a tool for generating scanners
Brought to you by:
wlestes
From: J. L. T. <jl...@ce...> - 2009-06-01 03:53:12
|
On Wednesday 27 May 2009 02:50:51 Samkit Jain wrote: > If I understand correctly then before execution of the command you would > know the valid (optional) tokens/endchar like '?'. Based on this > assumption, the rule can be made like: > > WORD [[:alnum:]_-]+ > > {WORD} { > if(isValidOption(yytext)) > return SPECIFIER; > else > return SPECIFICATION; > } > > In parser, use non-terminals like: > > command: main optional_specifier specification { ... } > ; > > main: /* handle main function and validations here */ > ; > > optional_specification: /* empty */ { ... } > > | SPECIFIER { ... } > > ; > > specification: /* handle specifications here */ > ; > > Regards, > Samkit Jain > The user who invokes "command" knows what value (s)he will use for endchar; the lexer and scanner must capture that value while processing the input stream, then recognize that value as the endchar token while processing the rest of the specifications. Leslie > On Wed, May 27, 2009 at 12:09 PM, Virgil Palanciuc < > > vir...@gm...> wrote: > > Hi, > > So, if I get it straight, your lexer should be something like this: > > > > <INITIAL>{command} { BEGIN(OPTIONAL_DELIMITER); /* do something for > > the command */ } > > <OPTIONAL_DELIMITER>{whitespace} { /* do nothing */ } > > <OPTIONAL_DELIMITER>. { // I'm using '.', assuming that the delimiter > > must be a single char; you can replace it with any other rule > > BEGIN(MORE_SPECIFICATIONS); // delimiter is optional, so > > we enter "MORE_SPECIFICATIONS" state regardless on whether this is indeed > > a delimiter char > > if( !isUserDelimiter(yytext) ) yyless(0); // if the > > char is not delimiter, then don't consume it > > else return TOK_DELIMITER; // if it is a > > delimiter, return the appropriate token to the parser. > > } > > > > <MORE_SPECIFICATIONS>{something} { /*.....*/; BEGIN(INITIAL) } > > > > Does this help/did I understand correctly what you are trying to do? > > Virgil. > > > > On Wed, May 27, 2009 at 7:40 AM, J. L. Turriff <jl...@ce...> > > > > wrote: > > > I'm setting up flex definitions for a scripting language, and > > > I've come > > > across an issue that is not addressed in the flex manual. The language > > > definition allows the user to define the value of an optional delimiter > > > token > > > at execution-time; the token has no default value. How do I handle > > > this > > > > in > > > > > the flex definitions? > > > > > > Here's an example of the usage: > > > > > > command (endchar ?) more specifications... > > > > > > The value of endchar has no default (its use is optional in > > > /more specifications/) but whatever value is supplied by the user (in > > > this case "?") must be recognized by the scanner. > > > > > > Leslie > > |