flex-help — help with using flex in other applications

Re: [Flex-help] Will the C++ scanner be removed?

[Will E. <wes...@gm...>]

No particular feature of the c++ scanner is set in stone as such. At
this point, removing it entirely is off the table, but it is a mess.

--Will


On Monday,  4 November 2013,  9:16 am -0800, Arthur Schwarz <asc...@at...> wrote:

> The following comment is output from either of the' flex source' 
> (with %option c++) command or the 'flex++ source' command. Is this 
> comment valid? Will you be removing the C++ scanner?
> 
> 
> >     /* The c++ scanner is a mess. The FlexLexer.h header file relies on the
> >      * following macro. This is required in order to pass the c++-multiple-scanners
> >      * test in the regression suite. We get reports that it breaks inheritance.
> >      * We will address this in a future release of flex, or omit the C++ scanner
> >      * altogether.
> >      */
> >     #define yyFlexLexer SlipFlexLexer
> 
> ------------------------------------------------------------------------------
> Android is increasing in popularity, but the open development platform that
> developers love is also attractive to malware creators. Download this white
> paper to learn more about secure code signing practices that can help keep
> Android apps secure.
> http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
> _______________________________________________
> Flex-help mailing list
> Fle...@li...
> https://lists.sourceforge.net/lists/listinfo/flex-help

-- 
Will Estes (wl...@us...)
Flex Project Maintainer
http://flex.sourceforge.net/

Re: [Flex-help] lex/yacc/gcc help

[Mark H. <ma...@co...>]

On 11/06/2013 03:19 AM, Akim Demaille wrote:
>
> Le 5 nov. 2013 ? 14:11, Mark Hounschell <ma...@co...> a Ècrit :
>
>> Thanks Akim,
>
> Hi!
>
>> Like I said in my original post, I am yacc/lex ignorant. The strange thing, that I haven't mentioned is that this all built successfully 4, 5, or more years ago on Linux.
>
> I would not have expected that, given what you sent.
>
>> I was recently able to find that executable that I had checked into CVS those years back and it works. I also found some notes of what I originally had to do to get it to build. All I had to do back then was:
>>
>> 1. Changed all ocurrances of yylval.ntype
>>                           to yylval.nvaltype.ntype
>> 2. Changed all occurances of yylval.nval
>>                           to yylval.nvaltype.nval
>>
>> 3. Changed all occurances of yybgin to yy_start
>>
>> 4. Changed all occurrences of "k =/ 10" to "k /= 10"
>
> Wow, that piece of software _is_ old!  Could you tell us what
> it is?
>

It's a micro code assembler for a legacy CPU boards micro engine. It is 
an early 80s generation CPU board. From back in the days when powerfull 
CPUs were not yet on a chip. I still repair these boards and often a 
micro diagnostic has to be modified to create tight scope loops. The 
peice of software I'm trying to build creates the assembler used to 
compile the micros and firmware for this board. There was a time this 
peice of software was "Company private". That was long ago though. I 
would have no problem providing the code if it would help me get it to 
build again?

>> As far as including the whole scanner in the yacc file, the y.tab.c file does include the lex file but at the very end of it. After this BEGIN shows up.
>
> BEGIN is a macro, so it must be #defined before it is
> used.
>
>> I assume lex.yy.c is the whole scanner?
>
> Yes, that's what I meant.
>
>> And I also assume you mean trying to include it before the BEGIN?
>
> To include it before the body of the parser.  You'll have to find
> the right place, as the scanner certainly also wants to "see" things
> from the parser, such as YYSTYPE I guess.
>

Since I don't remember what version of SuSE Linux I used when it 
actually did built right, I might just get out all my old dist CDs and 
see if I can pin down the versions that did work.

Regards
Mark

[Flex-help] Are Mulitple EOF's allowed

[Arthur S. <asc...@at...>]

The following code yields the following comment:


"gumdrop.lex:9: multiple <<EOF>> rules for start condition EOLCOMMENT"

If <EOLCOMMENT><<EOF>> is placed outside of the 

<EOLCOMMENT>{ scope the same error message is
given.

Is there any way of using multiple <<EOF>> patterns?
Section 12 End-Of-File Rules says that this is 

possible. Is there anything wrong with my code?

I don't understand what I can do and without some
way of detecting <<EOF>> patterns I have problems
with multiple input buffers/files since restoring
a lexer environment retains the last lexer state
if an included files. Therefore, returning to a
previous environment starts with the last state
of the replaced environment. If the replaced
file contains an unterminating, e.g., string or
comment, then returning to a previous environment
continues trying to detect a string or comment or
etc. This leads to some issues for me.

Do I have to use a kludge and always return to 

INITIAL when I restore an environment, and how
can I do that?

art



-------------- code --------------
%x EOLCOMMENT

%%
<<EOF>>   return 0;

<EOLCOMMENT>{
.** 
\n
<<EOF>> BEGIN(INITIAL);
}  /* <EOLCOMMENT> */

Re: [Flex-help] lex/yacc/gcc help

[Mark H. <ma...@co...>]

On 11/05/2013 04:43 AM, Akim Demaille wrote:
>
> Le 1 nov. 2013 ? 14:36, Mark Hounschell <ma...@co...> a Ècrit :
>
>> # make
>> ./translate5 cpu.def
>> cat yacc1.yinit yacctoks yacc2.yinit yaccrules yacc3.yinit > yacc.in
>> yacc -d yacc.in
>> cat lex1.linit lextext lexfield lex2.linit > lex.in
>> flex -l lex.in
>> gcc -ansi -w   y.tab.c -o cpu.asm -lgcc
>> yacc.in: In function ÎyyparseÌ:
>> yacc.in:2322:3: error: unknown type name ÎBEGINÌ
>
> BEGIN is something expected in a lex file, not in a Yacc
> file.  This is fishy.
>
>> case 207:
>> # line 2305 "yacc.in"
>> { if (yychar != -1) {
>>                         yyclearin;
>>                         while (yyleng > 0) {
>>                                 unput(yytext[--yyleng]); }}
>>                  BEGIN ctxn;} break;
>
> It looks like your Yacc parser is trying to influence your
> lex scanner.  In a properly designed scanner/parser couple,
> you should never see things such as "BEGIN" or "unput" in
> the parser: that's the scanner's job only.
>
> Either be brave and change all this, or maybe there is a way
> out, I don't know, by trying to #include the whole scanner in
> the yacc file.  In either case, expect pain :(
>
>

Thanks Akim,

Like I said in my original post, I am yacc/lex ignorant. The strange 
thing, that I haven't mentioned is that this all built successfully 4, 
5, or more years ago on Linux. I was recently able to find that 
executable that I had checked into CVS those years back and it works. I 
also found some notes of what I originally had to do to get it to build. 
All I had to do back then was:

1. Changed all ocurrances of yylval.ntype
                           to yylval.nvaltype.ntype
2. Changed all occurances of yylval.nval
                           to yylval.nvaltype.nval

3. Changed all occurances of yybgin to yy_start

4. Changed all occurances of "k =/ 10" to "k /= 10"

As far as including the whole scanner in the yacc file, the y.tab.c file 
does include the lex file but at the very end of it. After this BEGIN 
shows up.

End of y.tab.c:

/* Line 2041 of yacc.c  */
#line 2355 "yacc.in"


#include "mic.c"

#include "fields.c"
#include "lex.yy.c"

I assume lex.yy.c is the whole scanner? And I also assume you mean 
trying to include it before the BEGIN?

Regards
Mark

[Flex-help] Will the C++ scanner be removed?

[Arthur S. <asc...@at...>]

The following comment is output from either of the' flex source' 
(with %option c++) command or the 'flex++ source' command. Is this 
comment valid? Will you be removing the C++ scanner?


>     /* The c++ scanner is a mess. The FlexLexer.h header file relies on the
>      * following macro. This is required in order to pass the c++-multiple-scanners
>      * test in the regression suite. We get reports that it breaks inheritance.
>      * We will address this in a future release of flex, or omit the C++ scanner
>      * altogether.
>      */
>     #define yyFlexLexer SlipFlexLexer

[Flex-help] FLEX: States of the DFA as a program code

[Beenish R. <bee...@ho...>]

In some book, I read about flex that flex generates the states of the DFA as a program code instead of using table lookup. This can make the lexers fast but can use much more space than a table driven program.Now, I have two questions:What does this mean by states of DFA as a program code? How does this approach makes use of much more space as compared to table driven? 		 	   		  

Re: [Flex-help] lex/yacc/gcc help

[John P. H. <jph...@gm...>]

I'm saying that the flex variable set by the BEGIN macro is static.  
That is, it has file scope.  Thus, to reference this variable in a bison 
rule, you must include the flex output file in the first code block of 
the grammar file (x.y).  You'll also have to save the token list to a 
file so that it can be included in the first code block of the scanner 
file (x.l).

On 11/01/2013 08:56 PM, Mark Hounschell wrote:
> On 11/01/2013 10:35 AM, John P. Hartmann wrote:
>> BEGIN is a flex macro to set the scanner start state.  Changing the flex
>> state from the parser is frowned upon in some circles, but that is what
>> your yacc does.  So you need to imbed the flex generated code before the
>> parser. (Defining the macro won't buy you anything for it sets a static
>> variable in the flexx-generated C code.)
>>
>
> If I understand you correctly, your saying I need to replace  that 
> "BEGIN ctxn;" in that yacc file with the actual code that would be 
> generated if that was in one of the lex files? Somehow I think that's 
> not what you are saying.
>
> This is the entire lex file containing reference to ctxn. I'm not sure 
> which of the 2 are the ctxn code. The entire yacc file causing the 
> compile failure is already shown below.
>
> <code,ctxn>("#"[0-9a-fA-F]+|[0-9]+) {
>         if (yytext[0] == '#') {
>                 yylval.nvaltype.nval=htoi(yytext);
>                 if (yyleng > 9)
>                         fprintf(stderr, "Literal value '%s' out of 
> range at line %d\n",
>                                         yytext, linec);
>         } else
>                 yylval.nvaltype.nval=atoi(yytext);
>
>         yylval.nvaltype.ntype=2;
>         BEGIN code;
>         return(NUM);
> }
>
> <code>[ \t] {}
>
> ^[ \t]+ {BEGIN code;}
>
> <ctxn,code>[A-Za-z][\$\.A-Za-z0-9]* {BEGIN code; 
> return(p_symbol(yytext));}
>
> <code>\n {p_eol(); ++linec;}
>
> [ ,\t] {}
> \n      {p_eol(); ++linec;BEGIN 0;}
> . {return(YYERRCODE);}
>
>
> %%
>
> #include "process.c"
>
>
>> You might look into flex' exclusive start states (%x ctxn rather than %s
>> ctxn--lex only has the %s variety); they are likely to simplify your
>> flexer.
>>
>
> Lost me there. Remember I am lex/yacc ignorant. Would sending a 
> tarball of the code help?
>
> Mark
>
>> On 11/01/2013 02:36 PM, Mark Hounschell wrote:
>>> I am trying to port some code from an old unix box to Linux. It is a
>>> microcode compiler for an old CPU board. This code uses lex/yacc/cc.
>>> Obviously all works on this old unix box. I am completely lex/yacc
>>> ignorant but am not gcc ignorant. I am hoping someone can give me some
>>> advise as to what I need to do.
>>> I've subscribed and sent this email to both the flex-help and the
>>> help-bison mailing lists because I really have no idea what the problem
>>> really is. I hope that is appropiate.
>>>
>>> The Linux distribution OS is OpenSuSE-12.3 and the software versions in
>>> use are:
>>>
>>> gcc-4.7-7.1.1.i586
>>> glibc-2.17-4.7.1.i586
>>> flex-2.5.37-2.1.1.i586
>>> bison-2.6.5-2.1.1.i586
>>>
>>>
>>> The actual error I get from gcc when compiling this stuff on Linux is:
>>>
>>> # make
>>> ./translate5 cpu.def
>>> cat yacc1.yinit yacctoks yacc2.yinit yaccrules yacc3.yinit > yacc.in
>>> yacc -d yacc.in
>>> cat lex1.linit lextext lexfield lex2.linit > lex.in
>>> flex -l lex.in
>>> gcc -ansi -w   y.tab.c -o cpu.asm -lgcc
>>> yacc.in: In function ÎyyparseÌ:
>>> yacc.in:2322:3: error: unknown type name ÎBEGINÌ
>>> make: *** [cpu.asm] Error 1
>>>
>>> This is the output from the legacy "unix" machine:
>>>
>>> # make
>>> translate5 cpu.def
>>> cat yacc1.yinit yacctoks yacc2.yinit yaccrules yacc3.yinit > yacc.in
>>> /usr/bin/yacc -d yacc.in
>>> cat lex1.linit lextext lexfield lex2.linit > lex.in
>>> /usr/bin/lex lex.in
>>> 2277/3600 nodes(%e), 10911/17500 positions(%p), 1561/1900 (%n), 89797
>>> transitions
>>> , 136/170 packed char classes(%k), 3614/5000 packed transitions(%a),
>>> 2433/4000 output slots(%o)
>>> /bin/cc -z -B/lib/ -t0 y.tab.c -ly -ll -v -w -o cpu.asm
>>> /lib/cpp y.tab.c /tmp/ctm030464 -DON_SEL -Dselport -DGOULD_PN -DCOFF
>>> /lib/ccom /tmp/ctm030464 /tmp/ctm030463 -Xll -w
>>> /bin/as -o y.tab.o /tmp/ctm030463
>>> /bin/ld -X /lib/crt0.o -e start -o cpu.asm -z y.tab.o -ly -ll -lc
>>>
>>> Now from my limited mis/understanding the ÎBEGINÌ in the compile 
>>> failure
>>> above seems to be a "lex thing"? Yet it seems to appear in a yacc file?
>>> If I look at what ends up being in the y.tab.c file on both machines I
>>> see:
>>>
>>> Legacy unix box:
>>>
>>> case 207:
>>> # line 2305 "yacc.in"
>>> { if (yychar != -1) {
>>>                           yyclearin;
>>>                           while (yyleng > 0) {
>>>                                   unput(yytext[--yyleng]); }}
>>>                    BEGIN ctxn;} break;
>>>                   }
>>>                   goto yystack;  /* stack new state and value */
>>>
>>>           }
>>>
>>> The Linux box:
>>>
>>>     case 208:
>>> /* Line 1778 of yacc.c  */
>>> #line 2318 "yacc.in"
>>>       { if (yychar != -1) {
>>>                           yyclearin;
>>>                           while (yyleng > 0) {
>>>                                   unput(yytext[--yyleng]); }}
>>>                   BEGIN ctxn;}
>>>
>>>        break;
>>>
>>> Both the above come from the yacc3.yinit file.
>>>
>>> cnum    :       /*empty*/
>>>                   { if (yychar != -1) {
>>>                           yyclearin;
>>>                           while (yyleng > 0) {
>>>                                   unput(yytext[--yyleng]); }}
>>>                   BEGIN ctxn;}
>>>           ;
>>>
>>>
>>> The only references to "ctxn" I find are from a lex1.linit and
>>> lex2.linit files.
>>>
>>> lex1.linit:
>>>
>>> %C
>>>
>>>
>>> %e 3600
>>>
>>> %p 17500
>>>
>>> %a 5000
>>>
>>> %o 4000
>>> %n 1900
>>>
>>> %k 170
>>>
>>>
>>>
>>>           int statnum=0;
>>>           int externum=0;
>>>           int passc=0;
>>>           int linec=1;
>>>           int errc=0;
>>>           int eolflag = 0;
>>>           int codeflag = 0;
>>>           int lque = 0;
>>>           int plinec = 0;
>>>           int spacec = 1;
>>>           int pagesize = 55;
>>>           int skippage = 0;
>>>           int skipline = 0;
>>>           int pagecnt = 1;
>>>
>>>           int type1, type2, type3, type4, type5, type6, type7, type8;
>>>
>>>           int type9, typeA, typeB, typeC, typeD, typeE, typeF;
>>>
>>>
>>> %S ex  co  pb  lc  rz  bd  al eq va code
>>>
>>> %S rz2 bd2 sp hd hd2 hd3 hd4 dir ctxn
>>> .
>>> .
>>> .
>>> .
>>>
>>>
>>> lex2.linit:
>>>
>>> <code,ctxn>("#"[0-9a-fA-F]+|[0-9]+) {
>>>           if (yytext[0] == '#') {
>>>                   yylval.nvaltype.nval=htoi(yytext);
>>>                   if (yyleng > 9)
>>>                           fprintf(stderr, "Literal value '%s' out of
>>> range at line %d\n",
>>>                                           yytext, linec);
>>>           } else
>>>                   yylval.nvaltype.nval=atoi(yytext);
>>>
>>>           yylval.nvaltype.ntype=2;
>>>           BEGIN code;
>>>           return(NUM);
>>> }
>>>
>>> <code>[ \t] {}
>>>
>>> ^[ \t]+ {BEGIN code;}
>>>
>>> <ctxn,code>[A-Za-z][\$\.A-Za-z0-9]* {BEGIN code;
>>> return(p_symbol(yytext));}
>>>
>>> <code>\n {p_eol(); ++linec;}
>>>
>>> [ ,\t] {}
>>> \n      {p_eol(); ++linec;BEGIN 0;}
>>> . {return(YYERRCODE);}
>>>
>>>
>>> %%
>>>
>>> #include "process.c"
>>>
>>>
>>> Again, I am completely ignorant of lex and yac. Very strange and 
>>> cryptic
>>> looking code. I would greatly appreciate any pointers from anyone.
>>>
>>> Thanks in advance
>>> Regards
>>> Mark
>>
>>
>>
>> -------------------------------------------------------------
>> This email was processed through Xeams to filter junk messages.
>> If you feel this message has been tagged incorrectly, you can
>> change its category by clicking the link below. Click here
>> http://12.186.155.30:5272/FrontController?operation=mbeu&f=00001_-970_20131101_419884.eml&chkBayesian=1&pr=1&mt=1&ma=s 
>>
>> to mark email as junk.
>> -------------------------------------------------------------
>> .
>>

[Flex-help] Section 18 C++ compilation of C code incorrect

[Arthur S. <asc...@at...>]

Section 18


"The first way is to simply compile a scanner generated by flex 

using a C++ compiler instead of a C compiler.  You should not 

encounter any compilation errors (see Reporting Bugs)."


Compilation errors occur because flex input must be changed to
flex yyinput. Change the manual, the C code must be altered
to use yyinput() instead of input(). The flex statement which
causes this condition is given below.


#ifdef __cplusplus
    static int yyinput (void)
#else
    static int input  (void)
#endif

Re: [Flex-help] lex/yacc/gcc help

[Mark H. <ma...@co...>]

On 11/01/2013 10:35 AM, John P. Hartmann wrote:
> BEGIN is a flex macro to set the scanner start state.  Changing the flex
> state from the parser is frowned upon in some circles, but that is what
> your yacc does.  So you need to imbed the flex generated code before the
> parser. (Defining the macro won't buy you anything for it sets a static
> variable in the flexx-generated C code.)
>

If I understand you correctly, your saying I need to replace  that 
"BEGIN ctxn;" in that yacc file with the actual code that would be 
generated if that was in one of the lex files? Somehow I think that's 
not what you are saying.

This is the entire lex file containing reference to ctxn. I'm not sure 
which of the 2 are the ctxn code. The entire yacc file causing the 
compile failure is already shown below.

<code,ctxn>("#"[0-9a-fA-F]+|[0-9]+) {
         if (yytext[0] == '#') {
                 yylval.nvaltype.nval=htoi(yytext);
                 if (yyleng > 9)
                         fprintf(stderr, "Literal value '%s' out of 
range at line %d\n",
                                         yytext, linec);
         } else
                 yylval.nvaltype.nval=atoi(yytext);

         yylval.nvaltype.ntype=2;
         BEGIN code;
         return(NUM);
}

<code>[ \t] {}

^[ \t]+ {BEGIN code;}

<ctxn,code>[A-Za-z][\$\.A-Za-z0-9]* {BEGIN code; return(p_symbol(yytext));}

<code>\n {p_eol(); ++linec;}

[ ,\t] {}
\n      {p_eol(); ++linec;BEGIN 0;}
. {return(YYERRCODE);}


%%

#include "process.c"


> You might look into flex' exclusive start states (%x ctxn rather than %s
> ctxn--lex only has the %s variety); they are likely to simplify your
> flexer.
>

Lost me there. Remember I am lex/yacc ignorant. Would sending a tarball 
of the code help?

Mark

> On 11/01/2013 02:36 PM, Mark Hounschell wrote:
>> I am trying to port some code from an old unix box to Linux. It is a
>> microcode compiler for an old CPU board. This code uses lex/yacc/cc.
>> Obviously all works on this old unix box. I am completely lex/yacc
>> ignorant but am not gcc ignorant. I am hoping someone can give me some
>> advise as to what I need to do.
>> I've subscribed and sent this email to both the flex-help and the
>> help-bison mailing lists because I really have no idea what the problem
>> really is. I hope that is appropiate.
>>
>> The Linux distribution OS is OpenSuSE-12.3 and the software versions in
>> use are:
>>
>> gcc-4.7-7.1.1.i586
>> glibc-2.17-4.7.1.i586
>> flex-2.5.37-2.1.1.i586
>> bison-2.6.5-2.1.1.i586
>>
>>
>> The actual error I get from gcc when compiling this stuff on Linux is:
>>
>> # make
>> ./translate5 cpu.def
>> cat yacc1.yinit yacctoks yacc2.yinit yaccrules yacc3.yinit > yacc.in
>> yacc -d yacc.in
>> cat lex1.linit lextext lexfield lex2.linit > lex.in
>> flex -l lex.in
>> gcc -ansi -w   y.tab.c -o cpu.asm -lgcc
>> yacc.in: In function ÎyyparseÌ:
>> yacc.in:2322:3: error: unknown type name ÎBEGINÌ
>> make: *** [cpu.asm] Error 1
>>
>> This is the output from the legacy "unix" machine:
>>
>> # make
>> translate5 cpu.def
>> cat yacc1.yinit yacctoks yacc2.yinit yaccrules yacc3.yinit > yacc.in
>> /usr/bin/yacc -d yacc.in
>> cat lex1.linit lextext lexfield lex2.linit > lex.in
>> /usr/bin/lex lex.in
>> 2277/3600 nodes(%e), 10911/17500 positions(%p), 1561/1900 (%n), 89797
>> transitions
>> , 136/170 packed char classes(%k), 3614/5000 packed transitions(%a),
>> 2433/4000 output slots(%o)
>> /bin/cc -z -B/lib/ -t0 y.tab.c -ly -ll -v -w -o cpu.asm
>> /lib/cpp y.tab.c /tmp/ctm030464 -DON_SEL -Dselport -DGOULD_PN -DCOFF
>> /lib/ccom /tmp/ctm030464 /tmp/ctm030463 -Xll -w
>> /bin/as -o y.tab.o /tmp/ctm030463
>> /bin/ld -X /lib/crt0.o -e start -o cpu.asm -z y.tab.o -ly -ll -lc
>>
>> Now from my limited mis/understanding the ÎBEGINÌ in the compile failure
>> above seems to be a "lex thing"? Yet it seems to appear in a yacc file?
>> If I look at what ends up being in the y.tab.c file on both machines I
>> see:
>>
>> Legacy unix box:
>>
>> case 207:
>> # line 2305 "yacc.in"
>> { if (yychar != -1) {
>>                           yyclearin;
>>                           while (yyleng > 0) {
>>                                   unput(yytext[--yyleng]); }}
>>                    BEGIN ctxn;} break;
>>                   }
>>                   goto yystack;  /* stack new state and value */
>>
>>           }
>>
>> The Linux box:
>>
>>     case 208:
>> /* Line 1778 of yacc.c  */
>> #line 2318 "yacc.in"
>>       { if (yychar != -1) {
>>                           yyclearin;
>>                           while (yyleng > 0) {
>>                                   unput(yytext[--yyleng]); }}
>>                   BEGIN ctxn;}
>>
>>        break;
>>
>> Both the above come from the yacc3.yinit file.
>>
>> cnum    :       /*empty*/
>>                   { if (yychar != -1) {
>>                           yyclearin;
>>                           while (yyleng > 0) {
>>                                   unput(yytext[--yyleng]); }}
>>                   BEGIN ctxn;}
>>           ;
>>
>>
>> The only references to "ctxn" I find are from a lex1.linit and
>> lex2.linit files.
>>
>> lex1.linit:
>>
>> %C
>>
>>
>> %e 3600
>>
>> %p 17500
>>
>> %a 5000
>>
>> %o 4000
>> %n 1900
>>
>> %k 170
>>
>>
>>
>>           int statnum=0;
>>           int externum=0;
>>           int passc=0;
>>           int linec=1;
>>           int errc=0;
>>           int eolflag = 0;
>>           int codeflag = 0;
>>           int lque = 0;
>>           int plinec = 0;
>>           int spacec = 1;
>>           int pagesize = 55;
>>           int skippage = 0;
>>           int skipline = 0;
>>           int pagecnt = 1;
>>
>>           int type1, type2, type3, type4, type5, type6, type7, type8;
>>
>>           int type9, typeA, typeB, typeC, typeD, typeE, typeF;
>>
>>
>> %S ex  co  pb  lc  rz  bd  al eq va code
>>
>> %S rz2 bd2 sp hd hd2 hd3 hd4 dir ctxn
>> .
>> .
>> .
>> .
>>
>>
>> lex2.linit:
>>
>> <code,ctxn>("#"[0-9a-fA-F]+|[0-9]+) {
>>           if (yytext[0] == '#') {
>>                   yylval.nvaltype.nval=htoi(yytext);
>>                   if (yyleng > 9)
>>                           fprintf(stderr, "Literal value '%s' out of
>> range at line %d\n",
>>                                           yytext, linec);
>>           } else
>>                   yylval.nvaltype.nval=atoi(yytext);
>>
>>           yylval.nvaltype.ntype=2;
>>           BEGIN code;
>>           return(NUM);
>> }
>>
>> <code>[ \t] {}
>>
>> ^[ \t]+ {BEGIN code;}
>>
>> <ctxn,code>[A-Za-z][\$\.A-Za-z0-9]* {BEGIN code;
>> return(p_symbol(yytext));}
>>
>> <code>\n {p_eol(); ++linec;}
>>
>> [ ,\t] {}
>> \n      {p_eol(); ++linec;BEGIN 0;}
>> . {return(YYERRCODE);}
>>
>>
>> %%
>>
>> #include "process.c"
>>
>>
>> Again, I am completely ignorant of lex and yac. Very strange and cryptic
>> looking code. I would greatly appreciate any pointers from anyone.
>>
>> Thanks in advance
>> Regards
>> Mark
>
>
>
> -------------------------------------------------------------
> This email was processed through Xeams to filter junk messages.
> If you feel this message has been tagged incorrectly, you can
> change its category by clicking the link below. Click here
> http://12.186.155.30:5272/FrontController?operation=mbeu&f=00001_-970_20131101_419884.eml&chkBayesian=1&pr=1&mt=1&ma=s
> to mark email as junk.
> -------------------------------------------------------------
> .
>

[Flex-help] Multiple <<EOF>> error

[Arthur S. <asc...@at...>]

I have an <<EOF>> in <INITIAL> and in another start 
condition. This generates an error for multiple EOF's. 
If I remove the <<EOF>> in the non-INITIAL start 
condition then the error goes away. Is there any way
to have multiple <<EOF>>'s, and if not, how can a
non-INITIAL start condition be gracefully terminated
when an <<EOF>> is detected? Do I need to carry 
around a variable which says "I am in start condition
X" and place a case statement in the sole remaining
<<EOF>> to terminate?

art

[Flex-help] Example error in Section 10

[Arthur S. <asc...@at...>]

 <str>\\[0-7]{1,3} 
 <str>\\[0-9]+ 

The string description in Section 10 will not

work correctly. A number, e.g., 123, will be

considered an octal number (first rule matches)

and never a decimal number. Using the C/C++

definition for an octal number may be better.

 <str>\\0[0-7]{1,3} 
 <str>\\[1-9][0-9]+ 

This still has the defect that the rules allow
an indefinite sized decimal number. It might
be better to limit the number received for
decimal numbers or remove the restriction on
octal numbers for consistency.

 <str>\\0[0-7]{1,3} 
 <str>\\[1-9][0-9]{1,2}

And you do not include hexadecimal numbers.
Putting it all together and normalizing on
existing definitions of numbers gets us.

 <str>\\0[0-7]{1,3} 
 <str>\\[1-9][[:digit:]]{1,2}
 <str>\\x[[:xdigit:]]{1,2} 

We are left with the notion that \0 and all
that follow are octal numbers. \1 and all 
that follow are decimal numbers. \x and all
that follow are hexadecimal numbers. And
\xBEEF leads to a binary \xBE followed by
the characters EF. 

For somewhat different reasons you might want
to reconsider your comment example. In this
context what does '*', zero or more mean?

<comment>[^*\n]*        /* eat anything that's not a '*' */
<comment>"*"+[^*/\n]*   /* eat up '*'s not followed by '/'s */
<comment>\n             ++line_num;
<comment>"*"+"/"        BEGIN(comment_caller);

Another way of addressing comment recognition 
may be:
 

<comment>"*"+                      /* eat up all '*'    */
<comment>.*"*/"   BEGIN(INITIAL);  /* eat up everything */
<comment>[^*\n]+                   /* eat up everything except newline and '*' */
<comment>\n       ++line_num;      /* eat up newline    */

Which allows the manual to introduce the notion
that the longer match wins ("*"+ loses to .*"*/")
and that we really don't care what the comments
are (.*"*/") and that if the comment extends to
multiple lines ([^*\n]+) why that's ok.

The manual example seems to address cases that
address the semantics of a comment (where can
a '*' appear, what can follow it, etc.) The
second example focuses on what ends a comment
(*/) and comments which can extend across 
multiple lines ([*\n]+ and \n). 

It's all subjective, I admit, and counter-examples
don't win the day. But some editing of the
examples might be a good idea.

art

Re: [Flex-help] lex/yacc/gcc help

[Mark H. <ma...@co...>]

I notice you see in my email and responded with "eBEGINi". That must be 
a result of using colorgcc and the way it outputs gcc errors in color. 
That output was cut and pasted into the email. It is BEGIN. There is no 
"eBEGINi" anywhere in the code. And as I'm sure you know, BEGIN appears 
to be a FLEX keyword described in the FLEX manual Section 10 - Start 
Conditions. I have 2 Lex input files with MANY of these. And also the 
single yacc/bison file described below that seems to be what is not 
compiling on the Linux box, yet does on the legacy box..


On 11/01/2013 10:29 AM, Arthur Schwarz wrote:
> Do a grep of your source files to locate which file eBEGINi is located in. It does not appear in any of the
> files included in you e-mail.
>
>
>
> ----- Original Message -----
> From: Mark Hounschell <ma...@co...>
> To: hel...@gn...; fle...@li...
> Cc:
> Sent: Friday, November 1, 2013 6:36 AM
> Subject: [Flex-help] lex/yacc/gcc help
>
> I am trying to port some code from an old unix box to Linux. It is a
> microcode compiler for an old CPU board. This code uses lex/yacc/cc.
> Obviously all works on this old unix box. I am completely lex/yacc
> ignorant but am not gcc ignorant. I am hoping someone can give me some
> advise as to what I need to do.
> I've subscribed and sent this email to both the flex-help and the
> help-bison mailing lists because I really have no idea what the problem
> really is. I hope that is appropiate.
>
> The Linux distribution OS is OpenSuSE-12.3 and the software versions in
> use are:
>
> gcc-4.7-7.1.1.i586
> glibc-2.17-4.7.1.i586
> flex-2.5.37-2.1.1.i586
> bison-2.6.5-2.1.1.i586
>
>
> The actual error I get from gcc when compiling this stuff on Linux is:
>
> # make
> ./translate5 cpu.def
> cat yacc1.yinit yacctoks yacc2.yinit yaccrules yacc3.yinit > yacc.in
> yacc -d yacc.in
> cat lex1.linit lextext lexfield lex2.linit > lex.in
> flex -l lex.in
> gcc -ansi -w   y.tab.c -o cpu.asm -lgcc
> yacc.in: In function ÎyyparseÌ:
> yacc.in:2322:3: error: unknown type name ÎBEGINÌ
> make: *** [cpu.asm] Error 1
>
> This is the output from the legacy "unix" machine:
>
> # make
> translate5 cpu.def
> cat yacc1.yinit yacctoks yacc2.yinit yaccrules yacc3.yinit > yacc.in
> /usr/bin/yacc -d yacc.in
> cat lex1.linit lextext lexfield lex2.linit > lex.in
> /usr/bin/lex lex.in
> 2277/3600 nodes(%e), 10911/17500 positions(%p), 1561/1900 (%n), 89797
> transitions
> , 136/170 packed char classes(%k), 3614/5000 packed transitions(%a),
> 2433/4000 output slots(%o)
> /bin/cc -z -B/lib/ -t0 y.tab.c -ly -ll -v -w -o cpu.asm
> /lib/cpp y.tab.c /tmp/ctm030464 -DON_SEL -Dselport -DGOULD_PN -DCOFF
> /lib/ccom /tmp/ctm030464 /tmp/ctm030463 -Xll -w
> /bin/as -o y.tab.o /tmp/ctm030463
> /bin/ld -X /lib/crt0.o -e start -o cpu.asm -z y.tab.o -ly -ll -lc
>
> Now from my limited mis/understanding the ÎBEGINÌ in the compile failure
> above seems to be a "lex thing"? Yet it seems to appear in a yacc file?
> If I look at what ends up being in the y.tab.c file on both machines I see:
>
> Legacy unix box:
>
> case 207:
> # line 2305 "yacc.in"
> { if (yychar != -1) {
>                           yyclearin;
>                           while (yyleng > 0) {
>                                   unput(yytext[--yyleng]); }}
>                    BEGIN ctxn;} break;
>                   }
>                   goto yystack;  /* stack new state and value */
>
>           }
>
> The Linux box:
>
>     case 208:
> /* Line 1778 of yacc.c  */
> #line 2318 "yacc.in"
>       { if (yychar != -1) {
>                           yyclearin;
>                           while (yyleng > 0) {
>                                   unput(yytext[--yyleng]); }}
>                   BEGIN ctxn;}
>
>        break;
>
> Both the above come from the yacc3.yinit file.
>
> cnum    :       /*empty*/
>                   { if (yychar != -1) {
>                           yyclearin;
>                           while (yyleng > 0) {
>                                   unput(yytext[--yyleng]); }}
>                   BEGIN ctxn;}
>           ;
>
>
> The only references to "ctxn" I find are from a lex1.linit and
> lex2.linit files.
>
> lex1.linit:
>
> %C
>
>
> %e 3600
>
> %p 17500
>
> %a 5000
>
> %o 4000
> %n 1900
>
> %k 170
>
>
>
>           int statnum=0;
>           int externum=0;
>           int passc=0;
>           int linec=1;
>           int errc=0;
>           int eolflag = 0;
>           int codeflag = 0;
>           int lque = 0;
>           int plinec = 0;
>           int spacec = 1;
>           int pagesize = 55;
>           int skippage = 0;
>           int skipline = 0;
>           int pagecnt = 1;
>
>           int type1, type2, type3, type4, type5, type6, type7, type8;
>
>           int type9, typeA, typeB, typeC, typeD, typeE, typeF;
>
>
> %S ex  co  pb  lc  rz  bd  al eq va code
>
> %S rz2 bd2 sp hd hd2 hd3 hd4 dir ctxn
> .
> .
> .
> .
>
>
> lex2.linit:
>
> <code,ctxn>("#"[0-9a-fA-F]+|[0-9]+) {
>           if (yytext[0] == '#') {
>                   yylval.nvaltype.nval=htoi(yytext);
>                   if (yyleng > 9)
>                           fprintf(stderr, "Literal value '%s' out of
> range at line %d\n",
>                                           yytext, linec);
>           } else
>                   yylval.nvaltype.nval=atoi(yytext);
>
>           yylval.nvaltype.ntype=2;
>           BEGIN code;
>           return(NUM);
> }
>
> <code>[ \t] {}
>
> ^[ \t]+ {BEGIN code;}
>
> <ctxn,code>[A-Za-z][\$\.A-Za-z0-9]* {BEGIN code; return(p_symbol(yytext));}
>
> <code>\n {p_eol(); ++linec;}
>
> [ ,\t] {}
> \n      {p_eol(); ++linec;BEGIN 0;}
> . {return(YYERRCODE);}
>
>
> %%
>
> #include "process.c"
>
>
> Again, I am completely ignorant of lex and yac. Very strange and cryptic
> looking code. I would greatly appreciate any pointers from anyone.
>
> Thanks in advance
> Regards
> Mark
>
> ------------------------------------------------------------------------------
> Android is increasing in popularity, but the open development platform that
> developers love is also attractive to malware creators. Download this white
> paper to learn more about secure code signing practices that can help keep
> Android apps secure.
> http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
> _______________________________________________
> Flex-help mailing list
> Fle...@li...
> https://lists.sourceforge.net/lists/listinfo/flex-help
>
>
>
> -------------------------------------------------------------
> This email was processed through Xeams to filter junk messages.
> If you feel this message has been tagged incorrectly, you can
> change its category by clicking the link below.
> Click here http://12.186.155.30:5272/FrontController?operation=mbeu&f=00001_-930_20131101_419812.eml&chkBayesian=1&pr=1&mt=1&ma=s to mark email as junk.
> -------------------------------------------------------------
> .
>

Re: [Flex-help] lex/yacc/gcc help

[John P. H. <jph...@gm...>]

BEGIN is a flex macro to set the scanner start state.  Changing the flex 
state from the parser is frowned upon in some circles, but that is what 
your yacc does.  So you need to imbed the flex generated code before the 
parser. (Defining the macro won't buy you anything for it sets a static 
variable in the flexx-generated C code.)

You might look into flex' exclusive start states (%x ctxn rather than %s 
ctxn--lex only has the %s variety); they are likely to simplify your flexer.

On 11/01/2013 02:36 PM, Mark Hounschell wrote:
> I am trying to port some code from an old unix box to Linux. It is a
> microcode compiler for an old CPU board. This code uses lex/yacc/cc.
> Obviously all works on this old unix box. I am completely lex/yacc
> ignorant but am not gcc ignorant. I am hoping someone can give me some
> advise as to what I need to do.
> I've subscribed and sent this email to both the flex-help and the
> help-bison mailing lists because I really have no idea what the problem
> really is. I hope that is appropiate.
>
> The Linux distribution OS is OpenSuSE-12.3 and the software versions in
> use are:
>
> gcc-4.7-7.1.1.i586
> glibc-2.17-4.7.1.i586
> flex-2.5.37-2.1.1.i586
> bison-2.6.5-2.1.1.i586
>
>
> The actual error I get from gcc when compiling this stuff on Linux is:
>
> # make
> ./translate5 cpu.def
> cat yacc1.yinit yacctoks yacc2.yinit yaccrules yacc3.yinit > yacc.in
> yacc -d yacc.in
> cat lex1.linit lextext lexfield lex2.linit > lex.in
> flex -l lex.in
> gcc -ansi -w   y.tab.c -o cpu.asm -lgcc
> yacc.in: In function ëyyparseí:
> yacc.in:2322:3: error: unknown type name ëBEGINí
> make: *** [cpu.asm] Error 1
>
> This is the output from the legacy "unix" machine:
>
> # make
> translate5 cpu.def
> cat yacc1.yinit yacctoks yacc2.yinit yaccrules yacc3.yinit > yacc.in
> /usr/bin/yacc -d yacc.in
> cat lex1.linit lextext lexfield lex2.linit > lex.in
> /usr/bin/lex lex.in
> 2277/3600 nodes(%e), 10911/17500 positions(%p), 1561/1900 (%n), 89797
> transitions
> , 136/170 packed char classes(%k), 3614/5000 packed transitions(%a),
> 2433/4000 output slots(%o)
> /bin/cc -z -B/lib/ -t0 y.tab.c -ly -ll -v -w -o cpu.asm
> /lib/cpp y.tab.c /tmp/ctm030464 -DON_SEL -Dselport -DGOULD_PN -DCOFF
> /lib/ccom /tmp/ctm030464 /tmp/ctm030463 -Xll -w
> /bin/as -o y.tab.o /tmp/ctm030463
> /bin/ld -X /lib/crt0.o -e start -o cpu.asm -z y.tab.o -ly -ll -lc
>
> Now from my limited mis/understanding the ëBEGINí in the compile failure
> above seems to be a "lex thing"? Yet it seems to appear in a yacc file?
> If I look at what ends up being in the y.tab.c file on both machines I see:
>
> Legacy unix box:
>
> case 207:
> # line 2305 "yacc.in"
> { if (yychar != -1) {
>                           yyclearin;
>                           while (yyleng > 0) {
>                                   unput(yytext[--yyleng]); }}
>                    BEGIN ctxn;} break;
>                   }
>                   goto yystack;  /* stack new state and value */
>
>           }
>
> The Linux box:
>
>     case 208:
> /* Line 1778 of yacc.c  */
> #line 2318 "yacc.in"
>       { if (yychar != -1) {
>                           yyclearin;
>                           while (yyleng > 0) {
>                                   unput(yytext[--yyleng]); }}
>                   BEGIN ctxn;}
>
>        break;
>
> Both the above come from the yacc3.yinit file.
>
> cnum    :       /*empty*/
>                   { if (yychar != -1) {
>                           yyclearin;
>                           while (yyleng > 0) {
>                                   unput(yytext[--yyleng]); }}
>                   BEGIN ctxn;}
>           ;
>
>
> The only references to "ctxn" I find are from a lex1.linit and
> lex2.linit files.
>
> lex1.linit:
>
> %C
>
>
> %e 3600
>
> %p 17500
>
> %a 5000
>
> %o 4000
> %n 1900
>
> %k 170
>
>
>
>           int statnum=0;
>           int externum=0;
>           int passc=0;
>           int linec=1;
>           int errc=0;
>           int eolflag = 0;
>           int codeflag = 0;
>           int lque = 0;
>           int plinec = 0;
>           int spacec = 1;
>           int pagesize = 55;
>           int skippage = 0;
>           int skipline = 0;
>           int pagecnt = 1;
>
>           int type1, type2, type3, type4, type5, type6, type7, type8;
>
>           int type9, typeA, typeB, typeC, typeD, typeE, typeF;
>
>
> %S ex  co  pb  lc  rz  bd  al eq va code
>
> %S rz2 bd2 sp hd hd2 hd3 hd4 dir ctxn
> .
> .
> .
> .
>
>
> lex2.linit:
>
> <code,ctxn>("#"[0-9a-fA-F]+|[0-9]+) {
>           if (yytext[0] == '#') {
>                   yylval.nvaltype.nval=htoi(yytext);
>                   if (yyleng > 9)
>                           fprintf(stderr, "Literal value '%s' out of
> range at line %d\n",
>                                           yytext, linec);
>           } else
>                   yylval.nvaltype.nval=atoi(yytext);
>
>           yylval.nvaltype.ntype=2;
>           BEGIN code;
>           return(NUM);
> }
>
> <code>[ \t] {}
>
> ^[ \t]+ {BEGIN code;}
>
> <ctxn,code>[A-Za-z][\$\.A-Za-z0-9]* {BEGIN code; return(p_symbol(yytext));}
>
> <code>\n {p_eol(); ++linec;}
>
> [ ,\t] {}
> \n      {p_eol(); ++linec;BEGIN 0;}
> . {return(YYERRCODE);}
>
>
> %%
>
> #include "process.c"
>
>
> Again, I am completely ignorant of lex and yac. Very strange and cryptic
> looking code. I would greatly appreciate any pointers from anyone.
>
> Thanks in advance
> Regards
> Mark

Re: [Flex-help] lex/yacc/gcc help

[Arthur S. <asc...@at...>]

Do a grep of your source files to locate which file eBEGINi is located in. It does not appear in any of the files included in you e-mail.



----- Original Message -----
From: Mark Hounschell <ma...@co...>
To: hel...@gn...; fle...@li...
Cc: 
Sent: Friday, November 1, 2013 6:36 AM
Subject: [Flex-help] lex/yacc/gcc help

I am trying to port some code from an old unix box to Linux. It is a 
microcode compiler for an old CPU board. This code uses lex/yacc/cc. 
Obviously all works on this old unix box. I am completely lex/yacc 
ignorant but am not gcc ignorant. I am hoping someone can give me some 
advise as to what I need to do.
I've subscribed and sent this email to both the flex-help and the 
help-bison mailing lists because I really have no idea what the problem 
really is. I hope that is appropiate.

The Linux distribution OS is OpenSuSE-12.3 and the software versions in 
use are:

gcc-4.7-7.1.1.i586
glibc-2.17-4.7.1.i586
flex-2.5.37-2.1.1.i586
bison-2.6.5-2.1.1.i586


The actual error I get from gcc when compiling this stuff on Linux is:

# make
./translate5 cpu.def
cat yacc1.yinit yacctoks yacc2.yinit yaccrules yacc3.yinit > yacc.in
yacc -d yacc.in
cat lex1.linit lextext lexfield lex2.linit > lex.in
flex -l lex.in
gcc -ansi -w   y.tab.c -o cpu.asm -lgcc
yacc.in: In function ëyyparseí:
yacc.in:2322:3: error: unknown type name ëBEGINí
make: *** [cpu.asm] Error 1

This is the output from the legacy "unix" machine:

# make
translate5 cpu.def
cat yacc1.yinit yacctoks yacc2.yinit yaccrules yacc3.yinit > yacc.in
/usr/bin/yacc -d yacc.in
cat lex1.linit lextext lexfield lex2.linit > lex.in
/usr/bin/lex lex.in
2277/3600 nodes(%e), 10911/17500 positions(%p), 1561/1900 (%n), 89797 
transitions
, 136/170 packed char classes(%k), 3614/5000 packed transitions(%a), 
2433/4000 output slots(%o)
/bin/cc -z -B/lib/ -t0 y.tab.c -ly -ll -v -w -o cpu.asm
/lib/cpp y.tab.c /tmp/ctm030464 -DON_SEL -Dselport -DGOULD_PN -DCOFF
/lib/ccom /tmp/ctm030464 /tmp/ctm030463 -Xll -w
/bin/as -o y.tab.o /tmp/ctm030463
/bin/ld -X /lib/crt0.o -e start -o cpu.asm -z y.tab.o -ly -ll -lc

Now from my limited mis/understanding the ëBEGINí in the compile failure 
above seems to be a "lex thing"? Yet it seems to appear in a yacc file? 
If I look at what ends up being in the y.tab.c file on both machines I see:

Legacy unix box:

case 207:
# line 2305 "yacc.in"
{ if (yychar != -1) {
                         yyclearin;
                         while (yyleng > 0) {
                                 unput(yytext[--yyleng]); }}
                  BEGIN ctxn;} break;
                 }
                 goto yystack;  /* stack new state and value */

         }

The Linux box:

   case 208:
/* Line 1778 of yacc.c  */
#line 2318 "yacc.in"
     { if (yychar != -1) {
                         yyclearin;
                         while (yyleng > 0) {
                                 unput(yytext[--yyleng]); }}
                 BEGIN ctxn;}

      break;

Both the above come from the yacc3.yinit file.

cnum    :       /*empty*/
                 { if (yychar != -1) {
                         yyclearin;
                         while (yyleng > 0) {
                                 unput(yytext[--yyleng]); }}
                 BEGIN ctxn;}
         ;


The only references to "ctxn" I find are from a lex1.linit and 
lex2.linit files.

lex1.linit:

%C


%e 3600

%p 17500

%a 5000

%o 4000
%n 1900

%k 170



         int statnum=0;
         int externum=0;
         int passc=0;
         int linec=1;
         int errc=0;
         int eolflag = 0;
         int codeflag = 0;
         int lque = 0;
         int plinec = 0;
         int spacec = 1;
         int pagesize = 55;
         int skippage = 0;
         int skipline = 0;
         int pagecnt = 1;

         int type1, type2, type3, type4, type5, type6, type7, type8;

         int type9, typeA, typeB, typeC, typeD, typeE, typeF;


%S ex  co  pb  lc  rz  bd  al eq va code

%S rz2 bd2 sp hd hd2 hd3 hd4 dir ctxn
.
.
.
.


lex2.linit:

<code,ctxn>("#"[0-9a-fA-F]+|[0-9]+) {
         if (yytext[0] == '#') {
                 yylval.nvaltype.nval=htoi(yytext);
                 if (yyleng > 9)
                         fprintf(stderr, "Literal value '%s' out of 
range at line %d\n",
                                         yytext, linec);
         } else
                 yylval.nvaltype.nval=atoi(yytext);

         yylval.nvaltype.ntype=2;
         BEGIN code;
         return(NUM);
}

<code>[ \t] {}

^[ \t]+ {BEGIN code;}

<ctxn,code>[A-Za-z][\$\.A-Za-z0-9]* {BEGIN code; return(p_symbol(yytext));}

<code>\n {p_eol(); ++linec;}

[ ,\t] {}
\n      {p_eol(); ++linec;BEGIN 0;}
. {return(YYERRCODE);}


%%

#include "process.c"


Again, I am completely ignorant of lex and yac. Very strange and cryptic 
looking code. I would greatly appreciate any pointers from anyone.

Thanks in advance
Regards
Mark

------------------------------------------------------------------------------
Android is increasing in popularity, but the open development platform that
developers love is also attractive to malware creators. Download this white
paper to learn more about secure code signing practices that can help keep
Android apps secure.
http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
_______________________________________________
Flex-help mailing list
Fle...@li...
https://lists.sourceforge.net/lists/listinfo/flex-help

[Flex-help] lex/yacc/gcc help

[Mark H. <ma...@co...>]

I am trying to port some code from an old unix box to Linux. It is a 
microcode compiler for an old CPU board. This code uses lex/yacc/cc. 
Obviously all works on this old unix box. I am completely lex/yacc 
ignorant but am not gcc ignorant. I am hoping someone can give me some 
advise as to what I need to do.
I've subscribed and sent this email to both the flex-help and the 
help-bison mailing lists because I really have no idea what the problem 
really is. I hope that is appropiate.

The Linux distribution OS is OpenSuSE-12.3 and the software versions in 
use are:

gcc-4.7-7.1.1.i586
glibc-2.17-4.7.1.i586
flex-2.5.37-2.1.1.i586
bison-2.6.5-2.1.1.i586


The actual error I get from gcc when compiling this stuff on Linux is:

# make
./translate5 cpu.def
cat yacc1.yinit yacctoks yacc2.yinit yaccrules yacc3.yinit > yacc.in
yacc -d yacc.in
cat lex1.linit lextext lexfield lex2.linit > lex.in
flex -l lex.in
gcc -ansi -w   y.tab.c -o cpu.asm -lgcc
yacc.in: In function ëyyparseí:
yacc.in:2322:3: error: unknown type name ëBEGINí
make: *** [cpu.asm] Error 1

This is the output from the legacy "unix" machine:

# make
translate5 cpu.def
cat yacc1.yinit yacctoks yacc2.yinit yaccrules yacc3.yinit > yacc.in
/usr/bin/yacc -d yacc.in
cat lex1.linit lextext lexfield lex2.linit > lex.in
/usr/bin/lex lex.in
2277/3600 nodes(%e), 10911/17500 positions(%p), 1561/1900 (%n), 89797 
transitions
, 136/170 packed char classes(%k), 3614/5000 packed transitions(%a), 
2433/4000 output slots(%o)
/bin/cc -z -B/lib/ -t0 y.tab.c -ly -ll -v -w -o cpu.asm
/lib/cpp y.tab.c /tmp/ctm030464 -DON_SEL -Dselport -DGOULD_PN -DCOFF
/lib/ccom /tmp/ctm030464 /tmp/ctm030463 -Xll -w
/bin/as -o y.tab.o /tmp/ctm030463
/bin/ld -X /lib/crt0.o -e start -o cpu.asm -z y.tab.o -ly -ll -lc

Now from my limited mis/understanding the ëBEGINí in the compile failure 
above seems to be a "lex thing"? Yet it seems to appear in a yacc file? 
If I look at what ends up being in the y.tab.c file on both machines I see:

Legacy unix box:

case 207:
# line 2305 "yacc.in"
{ if (yychar != -1) {
                         yyclearin;
                         while (yyleng > 0) {
                                 unput(yytext[--yyleng]); }}
                  BEGIN ctxn;} break;
                 }
                 goto yystack;  /* stack new state and value */

         }

The Linux box:

   case 208:
/* Line 1778 of yacc.c  */
#line 2318 "yacc.in"
     { if (yychar != -1) {
                         yyclearin;
                         while (yyleng > 0) {
                                 unput(yytext[--yyleng]); }}
                 BEGIN ctxn;}

      break;

Both the above come from the yacc3.yinit file.

cnum    :       /*empty*/
                 { if (yychar != -1) {
                         yyclearin;
                         while (yyleng > 0) {
                                 unput(yytext[--yyleng]); }}
                 BEGIN ctxn;}
         ;


The only references to "ctxn" I find are from a lex1.linit and 
lex2.linit files.

lex1.linit:

%C


%e 3600

%p 17500

%a 5000

%o 4000
%n 1900

%k 170



         int statnum=0;
         int externum=0;
         int passc=0;
         int linec=1;
         int errc=0;
         int eolflag = 0;
         int codeflag = 0;
         int lque = 0;
         int plinec = 0;
         int spacec = 1;
         int pagesize = 55;
         int skippage = 0;
         int skipline = 0;
         int pagecnt = 1;

         int type1, type2, type3, type4, type5, type6, type7, type8;

         int type9, typeA, typeB, typeC, typeD, typeE, typeF;


%S ex  co  pb  lc  rz  bd  al eq va code

%S rz2 bd2 sp hd hd2 hd3 hd4 dir ctxn
.
.
.
.


lex2.linit:

<code,ctxn>("#"[0-9a-fA-F]+|[0-9]+) {
         if (yytext[0] == '#') {
                 yylval.nvaltype.nval=htoi(yytext);
                 if (yyleng > 9)
                         fprintf(stderr, "Literal value '%s' out of 
range at line %d\n",
                                         yytext, linec);
         } else
                 yylval.nvaltype.nval=atoi(yytext);

         yylval.nvaltype.ntype=2;
         BEGIN code;
         return(NUM);
}

<code>[ \t] {}

^[ \t]+ {BEGIN code;}

<ctxn,code>[A-Za-z][\$\.A-Za-z0-9]* {BEGIN code; return(p_symbol(yytext));}

<code>\n {p_eol(); ++linec;}

[ ,\t] {}
\n      {p_eol(); ++linec;BEGIN 0;}
. {return(YYERRCODE);}


%%

#include "process.c"


Again, I am completely ignorant of lex and yac. Very strange and cryptic 
looking code. I would greatly appreciate any pointers from anyone.

Thanks in advance
Regards
Mark

[Flex-help] User Code causes 2 mains

[Arthur S. <asc...@at...>]

First the good news. In Section 4 the second example works just fine. A single main() is created, and although I haven't executed it, I'm sure it works.

Now for the bad news. In my more complex example two main()s are created. This is because I have a "%option main" in my input file. 


Please document the restriction on the use of %option main and user code inclusion of a main, and if possible in one of the following releases of flex, provide a diagnostic message. (Or alternatively, tell me to read the manual, again.)


thanks
art

Re: [Flex-help] warning, rule cannot be matched

[Chris v. <che...@gm...>]

I think the default for flex is to understand the full range of 8-bit
characters.  (See the -7 and -8 switches.)

If by "generates an error message" you're talking about the "rule cannot be
matched" message, it makes sense to me.  [^[:alnum;]] will indeed match \n,
so the specific \n pattern is 'redundant' and will never get matched.

-Chris


On Sun, Oct 27, 2013 at 7:04 AM, Arthur Schwarz <asc...@at...>wrote:

> The little code snippet below generates an error message. The reason (I
> think) is that the set [^[:alnum:]] contains a '\n' within it. I had
> assumed that the character set involved was [:print:] = [\x20-\7E], which
> excludes all control characters, [:cntrl:] = [\x00-\x1F\x7F], but
> apparently this is not the case.
>
>
> What is the set of characters recognized by flex when an operation like
> this is performed? Is it[\x00-\x7F], [\x00-\xFF], or ?
>
>
> %option main
>
> %{
> #ifndef DEBUG
>   #define    DEBUG(stmt) stmt
> #else
>   #define    DEBUG(stmt)
> #endif
> %}
>
> %%
>
> [^[:alnum:]]    { DEBUG( ECHO; ) } /* [\x00-\x7F]{-}[:alnum:] ? */
> .               { DEBUG( ECHO; ) }
> \n              { DEBUG( ECHO; ) } /* warning, rule cannot be matched */
> <<EOF>>         { return 1; }
>
>
> ------------------------------------------------------------------------------
> October Webinars: Code for Performance
> Free Intel webinars can help you accelerate application performance.
> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most
> from
> the latest Intel processors and coprocessors. See abstracts and register >
> http://pubads.g.doubleclick.net/gampad/clk?id=60135991&iu=/4140/ostg.clktrk
> _______________________________________________
> Flex-help mailing list
> Fle...@li...
> https://lists.sourceforge.net/lists/listinfo/flex-help
>

Re: [Flex-help] Unsuccessfully trying to use {NAME}

[Chris v. <che...@gm...>]

I think this is a superset of your later question about comments messing up
named patterns, yes?

-Chris



On Sun, Oct 27, 2013 at 4:43 PM, Arthur Schwarz <asc...@at...>wrote:

> Sorry to be a pest (well, not really). I'm trying to use a neat feature
> that you have in flex, the ability to define a named regular expression in
> the definitions section and use it in the rules. Alas, all is lost. I've
> tried to follow the direction in Section 5.1 of the flex manual to no
> success. The code below shows an example of what I'm trying to do. It seems
> dead simple and I'm doing something dead wrong.
>
> >From Section 5.1 the expansion of {WHITESPACE} is "[ \a\b\f\r\t\v]" and
> {TRUE} is "([tT][rR][uU][eE])". Using the named expressions don't work.
> Substituting the regular expression is as a pattern the code works (except
> for executing the statements associated with '\n').
>
>
> Could you explain what is wrong with the code so that I can use named
> regular expressions?
>
> art
>
> %option main
>
> %{
> #include <stdio.h>
> %}
>
> WHITESPACE  [ \a\b\f\r\t\v]                        /* white space   */
> TRUE        [tT][rR][uU][eE]                       /* true          */
>
> %%
>   /*
>    * rule 1: flex complains "unrecognized rule"
>    * rule 2: flex complains "unrecognized rule"
>    * rule 3: No complaint and pattern is not recognized on input
>    * rule '[ \a\b\f\r\t\v]+' works
>    * rule '[tT][rR][uU][eE]' works
>    * rule '\n'               no output but seems to work
>    * rule '.'                works
>    */
>
>  /* rule 1  {WHITESPACE}+                    { printf("%35s %s\n",
> "{WHITESPACE}+", yytext); } */
>  /* rule 2  {TRUE}                           { printf("%35s %s\n",
> "{TRUE}", yytext); } */
>  /* rule 3  [{TRUE}]                         { printf("%35s %s\n",
> "{TRUE}", yytext); } */
> ([ \a\b\f\r\t\v]+)               { printf("%35s %s\n", "[
> \\a\\b\\f\\r\\t\\v]+", yytext); }
> ([tT][rR][uU][eE])               { printf("%35s %s\n", "[tT][rR][uU][eE]",
> yytext); }
> <<EOF>>                          { printf("%35s %s", "\n", "<<EOF>>" );
> return 1; }
> \n                               { printf("%35s %s", "\n", "" ); }
> .                                { printf("%35s %s\n", ".", yytext); }
>
>
> ------------------------------------------------------------------------------
> October Webinars: Code for Performance
> Free Intel webinars can help you accelerate application performance.
> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most
> from
> the latest Intel processors and coprocessors. See abstracts and register >
> http://pubads.g.doubleclick.net/gampad/clk?id=60135991&iu=/4140/ostg.clktrk
> _______________________________________________
> Flex-help mailing list
> Fle...@li...
> https://lists.sourceforge.net/lists/listinfo/flex-help
>

Re: [Flex-help] Where does the longest or first match apply?

[Chris v. <che...@gm...>]

Your code snippet works fine for me.  Or at least, it behaves the way you
expect, because the message I'm getting from flex is:

:5: warning, rule cannot be matched

Thus, the [[:space:]] rule is indeed winning.  I'm not sure why yours is
unrecognizing and fatally erroring..

I'm using flex 2.5.35 on redhat.

-Chris



On Tue, Oct 29, 2013 at 8:11 AM, Arthur Schwarz <asc...@at...>wrote:

> Section 7 says "When the generated scanner is run, it analyzes its input
> looking for strings which match any of its patterns. If it finds more than
> one match, it takes the one matching the most text (for trailing context
> rules, this includes the length of the trailing part, even though it will
> then be returned to the input). If it finds two or more matches of the same
> length, the rule listed first in the flex input file is chosen."
>
> The following code generates the following errors. Any idea why?
>
> [:space:] natches [ \f\n\r\t\v] and \n is a subset of this pattern. But
> using the 'longest' or the 'first' as a scanning rule the construct should
> be legal and in the case of a '\n' being seen the [[:space:]] rule should
> be chosen (the matches are equal, the first pattern is [[:space:]]).
>
> %option main
>
> %%
> [[:space:]]+
> \n
>
> :5: unrecognized rule
> :5: fatal parse error
>
>
>
> ------------------------------------------------------------------------------
> Android is increasing in popularity, but the open development platform that
> developers love is also attractive to malware creators. Download this white
> paper to learn more about secure code signing practices that can help keep
> Android apps secure.
> http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
> _______________________________________________
> Flex-help mailing list
> Fle...@li...
> https://lists.sourceforge.net/lists/listinfo/flex-help
>

Re: [Flex-help] Comments in the definition section

[Chris v. <che...@gm...>]

I'm pretty sure comments in flex have to be the only thing on the line.
It's not actually C, so you can't put comments after aliases.  Any line
that starts with whitespace is also implicitly considered a comment.  The
support for "/* .. */" block-commenting is just:
1. so you can have multiple lines of comments without having to start
multiple lines with leading whitespace, and
2. to give a visual indication of a block of comments, and
3. to allow you to comment a bunch of stuff out at once.

Or at least that's my recollection.  List lurkers, feel free to jump in if
I'm way off.

-Chris



On Tue, Oct 29, 2013 at 11:01 AM, Arthur Schwarz <asc...@at...>wrote:

> Section 5.1 "An unindented comment (i.e., a line
> beginning with ‘/*’) is copied verbatim to the output up
> to the next ‘*/’."
>
> Section 5.4 "Flex supports C-style comments, that is, anything between
> ‘/*’ and
> ‘*/’ is
> considered a comment."
>
> With the comment in place, the following diagnostic message is output.
> :13: unrecognized rule
>
> With the comment removed, the scanner is generated and compiles and works
> as expected.
>
> Is there anything wrong with the code?
>
> %option main
>
> %{
> #include <stdio.h>
> %}
> DECIMAL     [1-9][[:digit:]]*                      /* 1234          */
> %%
>      int col = 1;
>      int lineno = 1;
>
> ([[:space:]]{-}[\n])+       { col += yyleng; }
> \n                          { printf("%3d:%4d \\n\n", lineno, col);
> lineno++; col=1; }
> {DECIMAL}                   { printf("%3d:%4d DECIMAL        %s\n",
> lineno, col, yytext); col += yyleng; }
>
>
> ------------------------------------------------------------------------------
> Android is increasing in popularity, but the open development platform that
> developers love is also attractive to malware creators. Download this white
> paper to learn more about secure code signing practices that can help keep
> Android apps secure.
> http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
> _______________________________________________
> Flex-help mailing list
> Fle...@li...
> https://lists.sourceforge.net/lists/listinfo/flex-help
>

[Flex-help] Comments in the definition section

[Arthur S. <asc...@at...>]

Section 5.1 "An unindented comment (i.e., a line
beginning with ‘/*’) is copied verbatim to the output up
to the next ‘*/’."

Section 5.4 "Flex supports C-style comments, that is, anything between ‘/*’ and
‘*/’ is
considered a comment."

With the comment in place, the following diagnostic message is output.
:13: unrecognized rule

With the comment removed, the scanner is generated and compiles and works as expected.

Is there anything wrong with the code?

%option main

%{
#include <stdio.h>
%}
DECIMAL     [1-9][[:digit:]]*                      /* 1234          */
%%
     int col = 1;
     int lineno = 1;
     
([[:space:]]{-}[\n])+       { col += yyleng; }
\n                          { printf("%3d:%4d \\n\n", lineno, col); lineno++; col=1; }
{DECIMAL}                   { printf("%3d:%4d DECIMAL        %s\n", lineno, col, yytext); col += yyleng; }

[Flex-help] Where does the longest or first match apply?

[Arthur S. <asc...@at...>]

Section 7 says "When the generated scanner is run, it analyzes its input looking for strings which match any of its patterns. If it finds more than one match, it takes the one matching the most text (for trailing context rules, this includes the length of the trailing part, even though it will then be returned to the input). If it finds two or more matches of the same length, the rule listed first in the flex input file is chosen."

The following code generates the following errors. Any idea why?

[:space:] natches [ \f\n\r\t\v] and \n is a subset of this pattern. But using the 'longest' or the 'first' as a scanning rule the construct should be legal and in the case of a '\n' being seen the [[:space:]] rule should be chosen (the matches are equal, the first pattern is [[:space:]]).

%option main

%%
[[:space:]]+
\n

:5: unrecognized rule
:5: fatal parse error

[Flex-help] flex warning after processing 'C' code

[Arthur S. <asc...@at...>]

Is there any reason why flex processes 'C' code?

Removing (not commenting) the bottom pattern and flex generates a lexer.
Removing the top pattern and flex generates an error message.
Removing the top pattern and the 'C' code (everything between '{' and '}') and flex generates a lexer.

Note that the pattern in the top action is "[[:punct:]]" and in the bottom action is "[ [:punct:]]". The difference is that the top quoted pattern is not the same as the rule pattern for the top  rule but is the same as the rule pattern for the bottom rule.


The result is that flex processes 'C' code. Any reason that this is done? 


%option main

%{
#include <stdio.h>
%}

%%
                      /* works just fine */
[ [:punct:]] { printf("%35s %s\n", "[[:punct:]]", yytext); }  
                      /* Warning: excess arguments to builtin `m4_ifdef' ignored */
[ [:punct:]] { printf("%35s %s\n", "[ [:punct:]]", yytext);}

[Flex-help] Unsuccessfully trying to use {NAME}

[Arthur S. <asc...@at...>]

Sorry to be a pest (well, not really). I'm trying to use a neat feature that you have in flex, the ability to define a named regular expression in the definitions section and use it in the rules. Alas, all is lost. I've tried to follow the direction in Section 5.1 of the flex manual to no success. The code below shows an example of what I'm trying to do. It seems dead simple and I'm doing something dead wrong. 

From Section 5.1 the expansion of {WHITESPACE} is "[ \a\b\f\r\t\v]" and {TRUE} is "([tT][rR][uU][eE])". Using the named expressions don't work. Substituting the regular expression is as a pattern the code works (except for executing the statements associated with '\n').


Could you explain what is wrong with the code so that I can use named regular expressions?

art

%option main

%{
#include <stdio.h>
%}

WHITESPACE  [ \a\b\f\r\t\v]                        /* white space   */
TRUE        [tT][rR][uU][eE]                       /* true          */

%%
  /*
   * rule 1: flex complains "unrecognized rule"
   * rule 2: flex complains "unrecognized rule"
   * rule 3: No complaint and pattern is not recognized on input
   * rule '[ \a\b\f\r\t\v]+' works
   * rule '[tT][rR][uU][eE]' works
   * rule '\n'               no output but seems to work
   * rule '.'                works
   */

 /* rule 1  {WHITESPACE}+                    { printf("%35s %s\n", "{WHITESPACE}+", yytext); } */
 /* rule 2  {TRUE}                           { printf("%35s %s\n", "{TRUE}", yytext); } */
 /* rule 3  [{TRUE}]                         { printf("%35s %s\n", "{TRUE}", yytext); } */
([ \a\b\f\r\t\v]+)               { printf("%35s %s\n", "[ \\a\\b\\f\\r\\t\\v]+", yytext); }
([tT][rR][uU][eE])               { printf("%35s %s\n", "[tT][rR][uU][eE]", yytext); }
<<EOF>>                          { printf("%35s %s", "\n", "<<EOF>>" ); return 1; }
\n                               { printf("%35s %s", "\n", "" ); }
.                                { printf("%35s %s\n", ".", yytext); }

[Flex-help] warning, rule cannot be matched

[Arthur S. <asc...@at...>]

The little code snippet below generates an error message. The reason (I think) is that the set [^[:alnum:]] contains a '\n' within it. I had assumed that the character set involved was [:print:] = [\x20-\7E], which excludes all control characters, [:cntrl:] = [\x00-\x1F\x7F], but apparently this is not the case. 


What is the set of characters recognized by flex when an operation like this is performed? Is it[\x00-\x7F], [\x00-\xFF], or ?


%option main

%{
#ifndef DEBUG 
  #define    DEBUG(stmt) stmt
#else
  #define    DEBUG(stmt) 
#endif
%}

%%   

[^[:alnum:]]    { DEBUG( ECHO; ) } /* [\x00-\x7F]{-}[:alnum:] ? */
.               { DEBUG( ECHO; ) }
\n              { DEBUG( ECHO; ) } /* warning, rule cannot be matched */
<<EOF>>         { return 1; }