flex-help — help with using flex in other applications

Re: [Flex-help] yyget_lineno missing in lex.yy.c

[John G. <joh...@gm...>]

Thanks, Will. 

I don't have Ubuntu, so I'm just going off info I got from the user. We've been using yyget_lineno and friends for a while. It certainly works in 2.5.35 using the -l flag. 

I'll ask my colleagues about losing lex compat. mode. Not sure whether this is still really necessary for us. The original code for this project does date from the mid'80s, though. ;-)

John


> On Jul 6, 2016, at 6:22 PM, Will Estes <wes...@gm...> wrote:
> 
> You said Ubuntu 16.04 includes flex 2.5.4? That's ancient and the tokens you mention weren't added till after 2.5.4.
> 
> Is that right, though? Seems odd that Ubuntu would be shipping that old version of flex.
> 
> But the lex compatibility mode is your problem. The tokens you list are not a part of "lex"; they're a part of "flex". That is, they're in the list of things that gets left out if you say you want "lex".
> 
>> On Wednesday,  6 July 2016,  3:13 pm -0400, John Gibson <joh...@gm...> wrote:
>> 
>> Hi,
>> 
>> The project I work with (https://github.com/rtcmix/rtcmix) embeds a lexer built with flex. This builds correctly using flex 2.5.35 (macOS) and all previous versions. We use the lex compatibility mode flag (-l).
>> 
>> Recently, we received a report from a user on Ubuntu 16.04, which includes flex 2.5.4, about not being able to build our software. The problem is that the yyget_lineno, yyset_lineno, and yylex_destroy functions are not included in the lex.yy.c file generated by flex 2.5.4. Adding "%option yylineno" at the top of our .l file did not help -- not surprisingly, since the docs say that lex compat. mode already implies %option yylineno. We use no other options.
>> 
>> Is there something simple that I'm missing here? Has something changed that requires us to compile differently than we've had to before? Or is there something missing in this user's installation?
>> 
>> Thanks much,
>> John
>> 
>> 
>> ------------------------------------------------------------------------------
>> Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
>> Francisco, CA to explore cutting-edge tech and listen to tech luminaries
>> present their vision of the future. This family event has something for
>> everyone, including kids. Get more information and register today.
>> http://sdm.link/attshape
>> -- 
>> Flex-help mailing list
>> Fle...@li...
>> https://lists.sourceforge.net/lists/listinfo/flex-help
> 
> -- 
> Will Estes
> Flex Project Maintainer
> wes...@gm...
> https://github.com/westes/flex

Re: [Flex-help] yyget_lineno missing in lex.yy.c

[Will E. <wes...@gm...>]

You said Ubuntu 16.04 includes flex 2.5.4? That's ancient and the tokens you mention weren't added till after 2.5.4.

Is that right, though? Seems odd that Ubuntu would be shipping that old version of flex.

But the lex compatibility mode is your problem. The tokens you list are not a part of "lex"; they're a part of "flex". That is, they're in the list of things that gets left out if you say you want "lex".

On Wednesday,  6 July 2016,  3:13 pm -0400, John Gibson <joh...@gm...> wrote:

> Hi,
> 
> The project I work with (https://github.com/rtcmix/rtcmix) embeds a lexer built with flex. This builds correctly using flex 2.5.35 (macOS) and all previous versions. We use the lex compatibility mode flag (-l).
> 
> Recently, we received a report from a user on Ubuntu 16.04, which includes flex 2.5.4, about not being able to build our software. The problem is that the yyget_lineno, yyset_lineno, and yylex_destroy functions are not included in the lex.yy.c file generated by flex 2.5.4. Adding "%option yylineno" at the top of our .l file did not help -- not surprisingly, since the docs say that lex compat. mode already implies %option yylineno. We use no other options.
> 
> Is there something simple that I'm missing here? Has something changed that requires us to compile differently than we've had to before? Or is there something missing in this user's installation?
> 
> Thanks much,
> John
> 
> 
> ------------------------------------------------------------------------------
> Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
> Francisco, CA to explore cutting-edge tech and listen to tech luminaries
> present their vision of the future. This family event has something for
> everyone, including kids. Get more information and register today.
> http://sdm.link/attshape
> -- 
> Flex-help mailing list
> Fle...@li...
> https://lists.sourceforge.net/lists/listinfo/flex-help

-- 
Will Estes
Flex Project Maintainer
wes...@gm...
https://github.com/westes/flex

[Flex-help] yyget_lineno missing in lex.yy.c

[John G. <joh...@gm...>]

Hi,

The project I work with (https://github.com/rtcmix/rtcmix) embeds a lexer built with flex. This builds correctly using flex 2.5.35 (macOS) and all previous versions. We use the lex compatibility mode flag (-l).

Recently, we received a report from a user on Ubuntu 16.04, which includes flex 2.5.4, about not being able to build our software. The problem is that the yyget_lineno, yyset_lineno, and yylex_destroy functions are not included in the lex.yy.c file generated by flex 2.5.4. Adding "%option yylineno" at the top of our .l file did not help -- not surprisingly, since the docs say that lex compat. mode already implies %option yylineno. We use no other options.

Is there something simple that I'm missing here? Has something changed that requires us to compile differently than we've had to before? Or is there something missing in this user's installation?

Thanks much,
John

Re: [Flex-help] flex 2.6.0 lexer cannot be restarted on Solaris 11.3

[Mark B. <cha...@ya...>]

Ok apparently my old and crusty code written in 2002 ought to be modernised.  So instead of the old-fashioned close fd 0 and call fopen() immediately afterwards which I remember from time immemorial, if I use the fancy new shiny freopen() function with stdin instead it all turns out well and the problem goes away.
Still, interesting I thought as to why this works on Linux, but that's more of an academic question now.
Nothing to fix in flex then, just in the way that I redirect stdin.
Best regards,Mark.

Sent from Yahoo Mail on Android 
 
  On Fri, 10 Jun, 2016 at 9:31, Mark Bannister<cha...@ya...> wrote:   


>    I have all the equipment to test this and run through the whole 
> process but I know not many people do. I also know that Oracle is > pouring about a billion dollars into the new SPARC architecture and have > some new equipment about to release. Certainly the new SPARC M7 is a > monster and having 1024 cores at over 4GHz is pretty cool. Anyways, > Solaris is not going away any time soon and so it may be of value for me > to crack open a dev zone or some facilties for Will and some others to > use on these type of tasks.
I've done some more debugging and I've found the problem.  The issue is thatYY_INPUT is using fread() on stdin.  However, as we never fopen() or fclose()stdin, the file buffer is left hanging around.  I've tried various potential solutionsand the only one I can get to work is to use read() on STDIN_FILENO instead.That would require a change to the definition of the YY_INPUT macro.
Here is a test case:
#include <stdio.h>
#include <unistd.h>

#define MAX_BUFFER 100

int main(int argc, char **argv)
{
    FILE *fp;
    int i, n;
    char buf[MAX_BUFFER];

    if (argc<2) return -1;

    for (i=1; i<argc; i++)
    {
        close(STDIN_FILENO);
        if (!(fp=fopen(argv[i], "r"))) return -1;
        printf("%s open\n", argv[i]);
/*        n=(int)read(STDIN_FILENO, buf, MAX_BUFFER); */
        n=(int)fread(buf, 1, MAX_BUFFER, stdin);
        printf("%d bytes read\n", n);
        fclose(fp);
        printf("%s closed\n\n", argv[i]);
    }
    return 0;
}

Compile with gcc, create two single line files (file1 and file2) and compare theoutput on Linux vs. Solaris:
---- Linux (Fedora 23) ----$ ./testread file1 file2
file1 open
5 bytes read
file1 closed

file2 open
5 bytes read
file2 closed

---- Solaris 11.3 ----$ ./testread file1 file2
file1 open
5 bytes read
file1 closed

file2 open
0 bytes read
file2 closed
When run on Solaris, the stdin file buffer still records end of file when we really intendit to read from the second file.  It doesn't know that we have subsequently closedand re-opened the underlying file descriptor.
Also, if the first file were to be bigger than the read buffer (in this case more than 100bytes), when we think we're reading the second file we're actually reading the next100 bytes from the first file.
I've tried setvbuf() on stdin to set it to unbuffered, which makes no difference.  I've triedfflush(stdin) in various places, which also makes no difference.
The only fix I can find to work is to replace the fread() from stdin to a read() fromSTDIN_FILENO.  Then the behaviour on Solaris is identical to Linux, and the outputis as expected.

Best regards,Mark.
     

Re: [Flex-help] flex 2.6.0 lexer cannot be restarted on Solaris 11.3

[Mark B. <cha...@ya...>]

>    I have all the equipment to test this and run through the whole 
> process but I know not many people do. I also know that Oracle is > pouring about a billion dollars into the new SPARC architecture and have > some new equipment about to release. Certainly the new SPARC M7 is a > monster and having 1024 cores at over 4GHz is pretty cool. Anyways, > Solaris is not going away any time soon and so it may be of value for me > to crack open a dev zone or some facilties for Will and some others to > use on these type of tasks.
I've done some more debugging and I've found the problem.  The issue is thatYY_INPUT is using fread() on stdin.  However, as we never fopen() or fclose()stdin, the file buffer is left hanging around.  I've tried various potential solutionsand the only one I can get to work is to use read() on STDIN_FILENO instead.That would require a change to the definition of the YY_INPUT macro.
Here is a test case:
#include <stdio.h>
#include <unistd.h>

#define MAX_BUFFER 100

int main(int argc, char **argv)
{
    FILE *fp;
    int i, n;
    char buf[MAX_BUFFER];

    if (argc<2) return -1;

    for (i=1; i<argc; i++)
    {
        close(STDIN_FILENO);
        if (!(fp=fopen(argv[i], "r"))) return -1;
        printf("%s open\n", argv[i]);
/*        n=(int)read(STDIN_FILENO, buf, MAX_BUFFER); */
        n=(int)fread(buf, 1, MAX_BUFFER, stdin);
        printf("%d bytes read\n", n);
        fclose(fp);
        printf("%s closed\n\n", argv[i]);
    }
    return 0;
}

Compile with gcc, create two single line files (file1 and file2) and compare theoutput on Linux vs. Solaris:
---- Linux (Fedora 23) ----$ ./testread file1 file2
file1 open
5 bytes read
file1 closed

file2 open
5 bytes read
file2 closed

---- Solaris 11.3 ----$ ./testread file1 file2
file1 open
5 bytes read
file1 closed

file2 open
0 bytes read
file2 closed
When run on Solaris, the stdin file buffer still records end of file when we really intendit to read from the second file.  It doesn't know that we have subsequently closedand re-opened the underlying file descriptor.
Also, if the first file were to be bigger than the read buffer (in this case more than 100bytes), when we think we're reading the second file we're actually reading the next100 bytes from the first file.
I've tried setvbuf() on stdin to set it to unbuffered, which makes no difference.  I've triedfflush(stdin) in various places, which also makes no difference.
The only fix I can find to work is to replace the fread() from stdin to a read() fromSTDIN_FILENO.  Then the behaviour on Solaris is identical to Linux, and the outputis as expected.

Best regards,Mark.
  

Re: [Flex-help] flex 2.6.0 lexer cannot be restarted on Solaris 11.3

[Dennis C. <dc...@bl...>]

On 06/09/2016 10:23 AM, Mark Bannister wrote:
>
>
>
>> output from the make run would be helpful if you'd like assistance in dbugging this problem.
>
> Do you mean that you want to see flex -T output?  There's nothing else in the 'make' output of any interest.
> Thanks,Mark.


Sort of a reply to "list" and Will Estes :

    I have all the equipment to test this and run through the whole 
process but I know not many people do. I also know that Oracle is 
pouring about a billion dollars into the new SPARC architecture and have 
some new equipment about to release. Certainly the new SPARC M7 is a 
monster and having 1024 cores at over 4GHz is pretty cool. Anyways, 
Solaris is not going away any time soon and so it may be of value for me 
to crack open a dev zone or some facilties for Will and some others to 
use on these type of tasks.

     Also, I have been up to my ears with other $work but will get heads 
down on this one later today. I will start with Solaris 10 on x86_64 as 
well as SPARC baseline and then move upwards to the SPARC M7.

Dennis Clarke

Re: [Flex-help] flex 2.6.0 lexer cannot be restarted on Solaris 11.3

[Mark B. <cha...@ya...>]

> output from the make run would be helpful if you'd like assistance in dbugging this problem.

Do you mean that you want to see flex -T output?  There's nothing else in the 'make' output of any interest.
Thanks,Mark.
  

Re: [Flex-help] flex 2.6.0 lexer cannot be restarted on Solaris 11.3

[Will E. <wes...@gm...>]

output from the make run would be helpful if you'd like assistance in dbugging this problem.

On Wednesday,  8 June 2016, 11:45 pm -0400, Dennis Clarke <dc...@bl...> wrote:

> On 06/08/2016 03:53 PM, Mark Bannister wrote:
> > I installed Solaris 11.3 using the OVA file that Oracle provide for
> > VirtualBox, then:
> >
> > sudo pkg install gcc
> >
> > ... and run gmake from the extracted tar file I sent a link to earlier.
> >  No CFLAGS needed.
> 
> 
> OKay, so you are on some sort of x86 platform and you don't know the 
> compiler or anything else. Got it.
> 
> Dennis
> 
> 
> 
> ------------------------------------------------------------------------------
> What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
> patterns at an interface-level. Reveals which users, apps, and protocols are 
> consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
> J-Flow, sFlow and other flows. Make informed decisions using capacity 
> planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
> -- 
> Flex-help mailing list
> Fle...@li...
> https://lists.sourceforge.net/lists/listinfo/flex-help

-- 
Will Estes
Flex Project Maintainer
wes...@gm...
https://github.com/westes/flex

Re: [Flex-help] flex 2.6.0 lexer cannot be restarted on Solaris 11.3

[Mark B. <cha...@ya...>]

> > Sorry Dennis, that was a bit light on info, I see what you mean.  I'm using VirtualBox on Windows 10,

> > I'm on Intel, I'm using the latest version of gcc that Oracle have packaged (5.3.0) and I compiled without> > flags so that produces a 32-bit binary.  I will try with different versions of gcc next time I am near a computer.> > Apologies I gave you some misinformation there.  Now I'm on a computer I can see the gcc version I get when> I 'pkg install gcc' from the Solaris 11.3 build is 4.8.2.  I've tried on 4.7, no change.> > I'm collecting a bit more debug output from the test case I produced for this.
I didn't expect there to be a difference, but I tried installing flex and bison on Solaris and generating the parser and lexer there instead of relying on the C files created in Fedora, but no change.  They are different versions too (Fedora flex 2.6.0/bison 3.0.4, Solaris flex 2.5.35/bison 2.3).  So I'm seeing this with multiple versions of gcc and with multiple versions of flex.
I trimmed my test input files down to contain one line each: just the 'quit' command.
Using flex -d, this is the output from my test case on Fedora:
$ ./parse quit1 quit2
*** parsing quit1 line 1
--(end of buffer or a NUL)
--accepting rule at line 29 ("quit")
--accepting rule at line 23 ("
")
*** parsing quit1 line 2
--(end of buffer or a NUL)
--EOF (start condition 0)
*** parsing quit2 line 1
--(end of buffer or a NUL)
--accepting rule at line 29 ("quit")
--accepting rule at line 23 ("
")
*** parsing quit2 line 2
--(end of buffer or a NUL)
--EOF (start condition 0)


And this is the same on Solaris:
$ ./parse quit1 quit2
*** parsing quit1 line 1
--(end of buffer or a NUL)
--accepting rule at line 29 ("quit")
--accepting rule at line 23 ("
")
*** parsing quit1 line 2
--(end of buffer or a NUL)
--EOF (start condition 0)
*** parsing quit2 line 1
--(end of buffer or a NUL)
--EOF (start condition 0)
quit2 line 1: syntax error at ''


So when it breaks, I seem to be reaching the end of buffer without actually reading a single byte from the second file.  This is confirmed in truss output where I can see the file is opened to fd 0, but no reads occur at all, i.e.
2705:   open("quit2", O_RDONLY)                         = 0
2705:   write(2, " * * *   p a r s i n g  ", 12)        = 12
2705:   write(2, " q u i t 2", 5)                       = 5
2705:   write(2, "   l i n e  ", 6)                     = 6
2705:   write(2, " 1\n", 2)                             = 2
2705:   write(2, " - - ( e n d   o f   b u".., 27)      = 27
2705:   ioctl(0, TCGETA, 0xFEFFE820)                    Err#25 ENOTTY
2705:   write(2, " - - E O F   ( s t a r t".., 23)      = 23
2705:   write(2, " 0 )\n", 3)                           = 3
2705:   write(2, " q u i t 2", 5)                       = 5
2705:   write(2, "   l i n e  ", 6)                     = 6
2705:   write(2, " 1 :  ", 3)                           = 3
2705:   write(2, " s y n t a x   e r r o r", 12)        = 12
2705:   write(2, "   a t   '", 5)                       = 5
2705:   write(2, " '\n", 2)                             = 2
2705:   llseek(0, 0, SEEK_CUR)                          = 0
2705:   close(0)                                        = 0
2705:   llseek(0, 0, SEEK_CUR)                          Err#9 EBADF
2705:   _exit(1)

The reason I concluded this must be a problem with the lexer is because it is the lexer that is responsible for filling its own buffer.  So why does it think it's reached end of buffer on the second invocation, when in fact it hasn't actually read any new input?

Best regards,Mark.

  

Re: [Flex-help] flex 2.6.0 lexer cannot be restarted on Solaris 11.3

[Mark B. <cha...@ya...>]

> Sorry Dennis, that was a bit light on info, I see what you mean.  I'm using VirtualBox on Windows 10,> I'm on Intel, I'm using the latest version of gcc that Oracle have packaged (5.3.0) and I compiled without> flags so that produces a 32-bit binary.  I will try with different versions of gcc next time I am near a computer.
Apologies I gave you some misinformation there.  Now I'm on a computer I can see the gcc version I get whenI 'pkg install gcc' from the Solaris 11.3 build is 4.8.2.  I've tried on 4.7, no change.
I'm collecting a bit more debug output from the test case I produced for this.
Mark.

Re: [Flex-help] flex 2.6.0 lexer cannot be restarted on Solaris 11.3

[Mark B. <cha...@ya...>]

Sorry Dennis, that was a bit light on info, I see what you mean.  I'm using VirtualBox on Windows 10, I'm on Intel, I'm using the latest version of gcc that Oracle have packaged (5.3.0) and I compiled without flags so that produces a 32-bit binary.
I will try with different versions of gcc next time I am near a computer.
Best regards,Mark.

Sent from Yahoo Mail on Android 
 
  On Thu, 9 Jun, 2016 at 4:45, Dennis Clarke<dc...@bl...> wrote:   On 06/08/2016 03:53 PM, Mark Bannister wrote:
> I installed Solaris 11.3 using the OVA file that Oracle provide for
> VirtualBox, then:
>
> sudo pkg install gcc
>
> ... and run gmake from the extracted tar file I sent a link to earlier.
>  No CFLAGS needed.


OKay, so you are on some sort of x86 platform and you don't know the 
compiler or anything else. Got it.

Dennis


  

Re: [Flex-help] flex 2.6.0 lexer cannot be restarted on Solaris 11.3

[Dennis C. <dc...@bl...>]

On 06/08/2016 03:53 PM, Mark Bannister wrote:
> I installed Solaris 11.3 using the OVA file that Oracle provide for
> VirtualBox, then:
>
> sudo pkg install gcc
>
> ... and run gmake from the extracted tar file I sent a link to earlier.
>  No CFLAGS needed.


OKay, so you are on some sort of x86 platform and you don't know the 
compiler or anything else. Got it.

Dennis

Re: [Flex-help] flex 2.6.0 lexer cannot be restarted on Solaris 11.3

[Mark B. <cha...@ya...>]

I installed Solaris 11.3 using the OVA file that Oracle provide for VirtualBox, then:
sudo pkg install gcc
... and run gmake from the extracted tar file I sent a link to earlier.  No CFLAGS needed.
Best regards,Mark.
Sent from Yahoo Mail on Android 
 
  On Wed, 8 Jun, 2016 at 19:55, Dennis Clarke<dc...@bl...> wrote:   On 06/08/2016 01:40 PM, Mark Bannister wrote:
> It's all in the Makefile.

I don't have your makefile.

Just run cc -V and echo $CFLAGS.



  

Re: [Flex-help] flex 2.6.0 lexer cannot be restarted on Solaris 11.3

[Dennis C. <dc...@bl...>]

On 06/08/2016 01:40 PM, Mark Bannister wrote:
> It's all in the Makefile.

I don't have your makefile.

Just run cc -V and echo $CFLAGS.

Re: [Flex-help] flex 2.6.0 lexer cannot be restarted on Solaris 11.3

[Mark B. <cha...@ya...>]

It's all in the Makefile.

Sent from Yahoo Mail on Android 
 
  On Wed, 8 Jun, 2016 at 17:56, Dennis Clarke<dc...@bl...> wrote:   On 06/08/2016 12:19 PM, Mark Bannister wrote:
> It looks like a lexer issue to me. I've uploaded a simple test case here:
> https://sourceforge.net/projects/prose/files/prose/0.10.0/test-parser.tar.gz/download
>
> Compile with gmake on Solaris 11.3 and run:
> ./parse one two
> Compare with output from your favourite Linux.

What compiler did you use and what were your CFLAGS etc ?

Dennis



------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity 
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
-- 
Flex-help mailing list
Fle...@li...
https://lists.sourceforge.net/lists/listinfo/flex-help
  

Re: [Flex-help] flex 2.6.0 lexer cannot be restarted on Solaris 11.3

[Dennis C. <dc...@bl...>]

On 06/08/2016 12:19 PM, Mark Bannister wrote:
> It looks like a lexer issue to me. I've uploaded a simple test case here:
> https://sourceforge.net/projects/prose/files/prose/0.10.0/test-parser.tar.gz/download
>
> Compile with gmake on Solaris 11.3 and run:
> ./parse one two
> Compare with output from your favourite Linux.

What compiler did you use and what were your CFLAGS etc ?

Dennis

Re: [Flex-help] flex 2.6.0 lexer cannot be restarted on Solaris 11.3

[Mark B. <cha...@ya...>]

It looks like a lexer issue to me. I've uploaded a simple test case here:
https://sourceforge.net/projects/prose/files/prose/0.10.0/test-parser.tar.gz/download

Compile with gmake on Solaris 11.3 and run:
./parse one two
Compare with output from your favourite Linux.
Thanks,Mark.

Sent from Yahoo Mail on Android 
 
  On Tue, 7 Jun, 2016 at 4:48, Will Estes<wes...@gm...> wrote:   On Saturday,  4 June 2016,  7:28 am +0000, Mark Bannister <cha...@ya...> wrote:

> Hi,
> 
> 
> I'm having trouble with flex here (flex 2.6.0 and bison 3.0.4).  I can get it working fine on numerous Linux platforms, but it is misbehaving on Solaris 11.3 and I cannot get to the bottom of it.  The lexer is compiled on Fedora, then I do a make dist and copy the tarball to Solaris 11.3 and compile there, so flex (and bison) have run on Fedora not on Solaris.  However, this is what I am getting on an assembler tool of mine that is supposed to parse two separate input files, one after the other, by calling the lexer two times in succession:

Are you sure that the problem is in the lexer? Is it possible that it's in the parser?

Can you trim down the lexer and post the result so it's possible to see the code that's causing you trouble?

> $ prism -m test17 test17a
> prism: test17a.pal line 1: syntax error at ''
>  *** ERROR(S) OCCURRED DURING COMPILATION
> 
> Needless to say that test17.pal is not an empty file, and it assembles just fine on a Linux platform.  If I swap the filename arguments around:
> $ prism -m test17a test17
> prism: test17.pal line 1: syntax error at ''
>  *** ERROR(S) OCCURRED DURING COMPILATION
> 
> ... so the problem is occurring with the second file given to the lexer, whatever that file happens to be.  Now with some debug:
> $ prism -D50 -m test17 test17a<snip>Next token is token FUNC_RTN ()
> Shifting token FUNC_RTN ()
> Entering state 134
> Reading a token: Now at end of input.
> Reducing stack 0 by rule 519 (line 1647):
>    $1 = token FUNC_RTN ()
> -> $$ = nterm func_rtn ()
> Entering state 311
> Reducing stack 0 by rule 137 (line 245):
>    $1 = nterm func_rtn ()
> -> $$ = nterm line ()
> Entering state 851
> Reducing stack 0 by rule 2 (line 108):
>    $1 = nterm lines ()
>    $2 = nterm line ()
> -> $$ = nterm lines ()
> Entering state 175
> Now at end of input.
> Shifting token $end ()
> Entering state 850
> Cleanup: popping token $end ()
> Cleanup: popping nterm lines ()
> Starting parse
> Entering state 0
> Reading a token: Now at end of input.
> prism: test17a.pal line 1: syntax error at ''
> Cleanup: discarding lookahead token $end ()
>  *** ERROR(S) OCCURRED DURING COMPILATION
> 
> Compared with the same debug output on Linux:
> Next token is token FUNC_RTN ()
> Shifting token FUNC_RTN ()
> Entering state 134
> Reading a token: Now at end of input.
> Reducing stack 0 by rule 519 (line 1647):
>    $1 = token FUNC_RTN ()
> -> $$ = nterm func_rtn ()
> Entering state 311
> Reducing stack 0 by rule 137 (line 245):
>    $1 = nterm func_rtn ()
> -> $$ = nterm line ()
> Entering state 851
> Reducing stack 0 by rule 2 (line 108):
>    $1 = nterm lines ()
>    $2 = nterm line ()
> -> $$ = nterm lines ()
> Entering state 175
> Now at end of input.
> Shifting token $end ()
> Entering state 850
> Cleanup: popping token $end ()
> Cleanup: popping nterm lines ()
> Starting parse
> Entering state 0
> Reading a token: Next token is token LABEL ()
> Shifting token LABEL ()
> Entering state 167
> 
> So the debug isn't really helping me see the problem.  I've tried truss, but it isn't helping me either:
> $ truss -f -r all -w all -o /tmp/mb.truss ../../src/prism/prism -D50 -m test17 test17a
> Examining the output, I'm closing fd 0 at the end of processing the first file, then I open the next file on fd 0, but not a single byte is read from the second file.
> I have tried YY_FLUSH_BUFFER and BEGIN INITIAL after yyparse() returns, and I have also tried calling yylex_destroy() here too.  But there is no change.
> Please can someone help me determine which stubborn global is not being reset, and the correct way to reset it?
> The code is here: PROSE Programming Language / Mercurial / [5c41ac] /src/prism/lexer.l  ... due to the magic of SourceForge the web view has no line numbers, but look for the Pal_lexer() function and then where it calls yyparse().
> 
> |   |
> |   |   |   |   |   |
> | PROSE Programming Language / Mercurial / [5c41ac] /src/prism/lexer.l1679 lines (1511 with data), 44.4 kB  |
> |  |
> | View on sourceforge.net | Preview by Yahoo |
> |  |
> |   |
> 
> 
> ... it is called in a loop from here: PROSE Programming Language / Mercurial / [5c41ac] /src/prism/prism.c
> 
> |   |
> |   |   |   |   |   |
> | PROSE Programming Language / Mercurial / [5c41ac] /src/prism/prism.c/* ***************************************************************************** * * This source code can be downloaded from prose.sourceforge.net *  |
> |  |
> | View on sourceforge.net | Preview by Yahoo |
> |  |
> |   |
> 
> 
> Thanks in advance,Mark.
> 
> 
> 
> 
>  
> ------------------------------------------------------------------------------
> What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
> patterns at an interface-level. Reveals which users, apps, and protocols are 
> consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
> J-Flow, sFlow and other flows. Make informed decisions using capacity 
> planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
> -- 
> Flex-help mailing list
> Fle...@li...
> https://lists.sourceforge.net/lists/listinfo/flex-help

-- 
Will Estes
Flex Project Maintainer
wes...@gm...
https://github.com/westes/flex
  

Re: [Flex-help] flex 2.6.0 lexer cannot be restarted on Solaris 11.3

[Will E. <wes...@gm...>]

On Saturday,  4 June 2016,  7:28 am +0000, Mark Bannister <cha...@ya...> wrote:

> Hi,
> 
> 
> I'm having trouble with flex here (flex 2.6.0 and bison 3.0.4).  I can get it working fine on numerous Linux platforms, but it is misbehaving on Solaris 11.3 and I cannot get to the bottom of it.  The lexer is compiled on Fedora, then I do a make dist and copy the tarball to Solaris 11.3 and compile there, so flex (and bison) have run on Fedora not on Solaris.  However, this is what I am getting on an assembler tool of mine that is supposed to parse two separate input files, one after the other, by calling the lexer two times in succession:

Are you sure that the problem is in the lexer? Is it possible that it's in the parser?

Can you trim down the lexer and post the result so it's possible to see the code that's causing you trouble?

> $ prism -m test17 test17a
> prism: test17a.pal line 1: syntax error at ''
>  *** ERROR(S) OCCURRED DURING COMPILATION
> 
> Needless to say that test17.pal is not an empty file, and it assembles just fine on a Linux platform.  If I swap the filename arguments around:
> $ prism -m test17a test17
> prism: test17.pal line 1: syntax error at ''
>  *** ERROR(S) OCCURRED DURING COMPILATION
> 
> ... so the problem is occurring with the second file given to the lexer, whatever that file happens to be.  Now with some debug:
> $ prism -D50 -m test17 test17a<snip>Next token is token FUNC_RTN ()
> Shifting token FUNC_RTN ()
> Entering state 134
> Reading a token: Now at end of input.
> Reducing stack 0 by rule 519 (line 1647):
>    $1 = token FUNC_RTN ()
> -> $$ = nterm func_rtn ()
> Entering state 311
> Reducing stack 0 by rule 137 (line 245):
>    $1 = nterm func_rtn ()
> -> $$ = nterm line ()
> Entering state 851
> Reducing stack 0 by rule 2 (line 108):
>    $1 = nterm lines ()
>    $2 = nterm line ()
> -> $$ = nterm lines ()
> Entering state 175
> Now at end of input.
> Shifting token $end ()
> Entering state 850
> Cleanup: popping token $end ()
> Cleanup: popping nterm lines ()
> Starting parse
> Entering state 0
> Reading a token: Now at end of input.
> prism: test17a.pal line 1: syntax error at ''
> Cleanup: discarding lookahead token $end ()
>  *** ERROR(S) OCCURRED DURING COMPILATION
> 
> Compared with the same debug output on Linux:
> Next token is token FUNC_RTN ()
> Shifting token FUNC_RTN ()
> Entering state 134
> Reading a token: Now at end of input.
> Reducing stack 0 by rule 519 (line 1647):
>    $1 = token FUNC_RTN ()
> -> $$ = nterm func_rtn ()
> Entering state 311
> Reducing stack 0 by rule 137 (line 245):
>    $1 = nterm func_rtn ()
> -> $$ = nterm line ()
> Entering state 851
> Reducing stack 0 by rule 2 (line 108):
>    $1 = nterm lines ()
>    $2 = nterm line ()
> -> $$ = nterm lines ()
> Entering state 175
> Now at end of input.
> Shifting token $end ()
> Entering state 850
> Cleanup: popping token $end ()
> Cleanup: popping nterm lines ()
> Starting parse
> Entering state 0
> Reading a token: Next token is token LABEL ()
> Shifting token LABEL ()
> Entering state 167
> 
> So the debug isn't really helping me see the problem.  I've tried truss, but it isn't helping me either:
> $ truss -f -r all -w all -o /tmp/mb.truss ../../src/prism/prism -D50 -m test17 test17a
> Examining the output, I'm closing fd 0 at the end of processing the first file, then I open the next file on fd 0, but not a single byte is read from the second file.
> I have tried YY_FLUSH_BUFFER and BEGIN INITIAL after yyparse() returns, and I have also tried calling yylex_destroy() here too.  But there is no change.
> Please can someone help me determine which stubborn global is not being reset, and the correct way to reset it?
> The code is here: PROSE Programming Language / Mercurial / [5c41ac] /src/prism/lexer.l  ... due to the magic of SourceForge the web view has no line numbers, but look for the Pal_lexer() function and then where it calls yyparse().
> 
> |   |
> |   |   |   |   |   |
> | PROSE Programming Language / Mercurial / [5c41ac] /src/prism/lexer.l1679 lines (1511 with data), 44.4 kB  |
> |  |
> | View on sourceforge.net | Preview by Yahoo |
> |  |
> |   |
> 
> 
> ... it is called in a loop from here: PROSE Programming Language / Mercurial / [5c41ac] /src/prism/prism.c
> 
> |   |
> |   |   |   |   |   |
> | PROSE Programming Language / Mercurial / [5c41ac] /src/prism/prism.c/* ***************************************************************************** * * This source code can be downloaded from prose.sourceforge.net *  |
> |  |
> | View on sourceforge.net | Preview by Yahoo |
> |  |
> |   |
> 
> 
> Thanks in advance,Mark.
> 
> 
> 
> 
>   
> ------------------------------------------------------------------------------
> What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
> patterns at an interface-level. Reveals which users, apps, and protocols are 
> consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
> J-Flow, sFlow and other flows. Make informed decisions using capacity 
> planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
> -- 
> Flex-help mailing list
> Fle...@li...
> https://lists.sourceforge.net/lists/listinfo/flex-help

-- 
Will Estes
Flex Project Maintainer
wes...@gm...
https://github.com/westes/flex

Re: [Flex-help] flex 2.6.0 lexer cannot be restarted on Solaris 11.3

[Will E. <wes...@gm...>]

Flex strives to be POSIX compatible. That leaves some areas that are either ambiguous or, where POSIX is "old", some things that are not as relevant as they once were.

That being said, we want flex to be portable when that's reasonable and good.

On Sunday,  5 June 2016,  4:08 pm -0400, Dennis Clarke <dc...@bl...> wrote:

> On 06/04/2016 03:28 AM, Mark Bannister wrote:
> > Hi,
> >
> >
> > I'm having trouble with flex here (flex 2.6.0 and bison 3.0.4).
> 
> Firstly .. Solaris is not Linux. It is UNIX(tm).  So expect just about 
> anything non-portable from the linux world to be a struggle. However 
> flex is pretty darn good software.
> 
> So I'll go get the sources and have a look on a baseline Solaris 10 
> system and then move upwards to a SPARC M7 based Sol 11.3 system and 
> then get back to you on this.
> 
> Dennis
> 
> 
> 
> 
> ------------------------------------------------------------------------------
> What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
> patterns at an interface-level. Reveals which users, apps, and protocols are 
> consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
> J-Flow, sFlow and other flows. Make informed decisions using capacity 
> planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
> -- 
> Flex-help mailing list
> Fle...@li...
> https://lists.sourceforge.net/lists/listinfo/flex-help

-- 
Will Estes
Flex Project Maintainer
wes...@gm...
https://github.com/westes/flex

Re: [Flex-help] flex 2.6.0 lexer cannot be restarted on Solaris 11.3

[Mark B. <cha...@ya...>]

Hi Dennis,
Thanks for your reply, that would be very helpful.  Yes I know Solaris is not Linux :-) My project has been compiling successfully on Solaris since 2002 and I've always used flex and bison.
I changed two things recently: I upgraded from flex 2.5.4 and I compiled on Solaris 11.3 for the first time. It compiled on Solaris 11.1 fine in the past (and earlier versions of Solaris).  I tried reverting back to flex 2.5.4 but I get the same problem.  Not sure how to debug this any further?
Best regards,Mark.

 
 
  On Sun, 5 Jun, 2016 at 21:08, Dennis Clarke<dc...@bl...> wrote:   On 06/04/2016 03:28 AM, Mark Bannister wrote:
> Hi,
>
>
> I'm having trouble with flex here (flex 2.6.0 and bison 3.0.4).

Firstly .. Solaris is not Linux. It is UNIX(tm).  So expect just about 
anything non-portable from the linux world to be a struggle. However 
flex is pretty darn good software.

So I'll go get the sources and have a look on a baseline Solaris 10 
system and then move upwards to a SPARC M7 based Sol 11.3 system and 
then get back to you on this.

Dennis




------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity 
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
-- 
Flex-help mailing list
Fle...@li...
https://lists.sourceforge.net/lists/listinfo/flex-help
  

Re: [Flex-help] flex 2.6.0 lexer cannot be restarted on Solaris 11.3

[Dennis C. <dc...@bl...>]

On 06/04/2016 03:28 AM, Mark Bannister wrote:
> Hi,
>
>
> I'm having trouble with flex here (flex 2.6.0 and bison 3.0.4).

Firstly .. Solaris is not Linux. It is UNIX(tm).  So expect just about 
anything non-portable from the linux world to be a struggle. However 
flex is pretty darn good software.

So I'll go get the sources and have a look on a baseline Solaris 10 
system and then move upwards to a SPARC M7 based Sol 11.3 system and 
then get back to you on this.

Dennis

[Flex-help] flex 2.6.0 lexer cannot be restarted on Solaris 11.3

[Mark B. <cha...@ya...>]

Hi,


I'm having trouble with flex here (flex 2.6.0 and bison 3.0.4).  I can get it working fine on numerous Linux platforms, but it is misbehaving on Solaris 11.3 and I cannot get to the bottom of it.  The lexer is compiled on Fedora, then I do a make dist and copy the tarball to Solaris 11.3 and compile there, so flex (and bison) have run on Fedora not on Solaris.  However, this is what I am getting on an assembler tool of mine that is supposed to parse two separate input files, one after the other, by calling the lexer two times in succession:
$ prism -m test17 test17a
prism: test17a.pal line 1: syntax error at ''
 *** ERROR(S) OCCURRED DURING COMPILATION

Needless to say that test17.pal is not an empty file, and it assembles just fine on a Linux platform.  If I swap the filename arguments around:
$ prism -m test17a test17
prism: test17.pal line 1: syntax error at ''
 *** ERROR(S) OCCURRED DURING COMPILATION

... so the problem is occurring with the second file given to the lexer, whatever that file happens to be.  Now with some debug:
$ prism -D50 -m test17 test17a<snip>Next token is token FUNC_RTN ()
Shifting token FUNC_RTN ()
Entering state 134
Reading a token: Now at end of input.
Reducing stack 0 by rule 519 (line 1647):
   $1 = token FUNC_RTN ()
-> $$ = nterm func_rtn ()
Entering state 311
Reducing stack 0 by rule 137 (line 245):
   $1 = nterm func_rtn ()
-> $$ = nterm line ()
Entering state 851
Reducing stack 0 by rule 2 (line 108):
   $1 = nterm lines ()
   $2 = nterm line ()
-> $$ = nterm lines ()
Entering state 175
Now at end of input.
Shifting token $end ()
Entering state 850
Cleanup: popping token $end ()
Cleanup: popping nterm lines ()
Starting parse
Entering state 0
Reading a token: Now at end of input.
prism: test17a.pal line 1: syntax error at ''
Cleanup: discarding lookahead token $end ()
 *** ERROR(S) OCCURRED DURING COMPILATION

Compared with the same debug output on Linux:
Next token is token FUNC_RTN ()
Shifting token FUNC_RTN ()
Entering state 134
Reading a token: Now at end of input.
Reducing stack 0 by rule 519 (line 1647):
   $1 = token FUNC_RTN ()
-> $$ = nterm func_rtn ()
Entering state 311
Reducing stack 0 by rule 137 (line 245):
   $1 = nterm func_rtn ()
-> $$ = nterm line ()
Entering state 851
Reducing stack 0 by rule 2 (line 108):
   $1 = nterm lines ()
   $2 = nterm line ()
-> $$ = nterm lines ()
Entering state 175
Now at end of input.
Shifting token $end ()
Entering state 850
Cleanup: popping token $end ()
Cleanup: popping nterm lines ()
Starting parse
Entering state 0
Reading a token: Next token is token LABEL ()
Shifting token LABEL ()
Entering state 167

So the debug isn't really helping me see the problem.  I've tried truss, but it isn't helping me either:
$ truss -f -r all -w all -o /tmp/mb.truss ../../src/prism/prism -D50 -m test17 test17a
Examining the output, I'm closing fd 0 at the end of processing the first file, then I open the next file on fd 0, but not a single byte is read from the second file.
I have tried YY_FLUSH_BUFFER and BEGIN INITIAL after yyparse() returns, and I have also tried calling yylex_destroy() here too.  But there is no change.
Please can someone help me determine which stubborn global is not being reset, and the correct way to reset it?
The code is here: PROSE Programming Language / Mercurial / [5c41ac] /src/prism/lexer.l  ... due to the magic of SourceForge the web view has no line numbers, but look for the Pal_lexer() function and then where it calls yyparse().

|   |
|   |   |   |   |   |
| PROSE Programming Language / Mercurial / [5c41ac] /src/prism/lexer.l1679 lines (1511 with data), 44.4 kB  |
|  |
| View on sourceforge.net | Preview by Yahoo |
|  |
|   |


... it is called in a loop from here: PROSE Programming Language / Mercurial / [5c41ac] /src/prism/prism.c

|   |
|   |   |   |   |   |
| PROSE Programming Language / Mercurial / [5c41ac] /src/prism/prism.c/* ***************************************************************************** * * This source code can be downloaded from prose.sourceforge.net *  |
|  |
| View on sourceforge.net | Preview by Yahoo |
|  |
|   |


Thanks in advance,Mark.




  

Re: [Flex-help] Some flex patch snippets for Solaris.

[Dennis C. <dc...@bl...>]

>
>>
>> As for finding m4, the solution needs to be more general. While what you do works for people who want to use the stock setup on Solaris, I can easily see other situations where it doesn't do enough.
>


Does env var M4=/usr/local/bin/m4 not work ?


dc

Re: [Flex-help] Some flex patch snippets for Solaris.

[Will E. <wes...@gm...>]

Glad the two changes that are in the post-2.6.1 tree are helpful. I have plans to put out a 2.6.2 in the near future. It's not scheduled yet, but "in the next couple weeks" seems the most likely time frame.

On Sunday, 22 May 2016,  7:48 am -0700, Rich Burridge <ric...@or...> wrote:

> On 05/20/16 03:12 PM, Will Estes wrote:
> > The bootstrapping of flex was reworked after 2.6.1 was released. I -think- it's immune to the vpath build that you do. If you can verify that against the current git tree, let me know the results. Ditto the building of the man page.
> 
> Confirmed. Both are now fine.
> 
> >
> > As for finding m4, the solution needs to be more general. While what you do works for people who want to use the stock setup on Solaris, I can easily see other situations where it doesn't do enough.
> 
> Agreed.
> 
> >
> > I'll open an issue to describe the problem.
> >
> > What should happen is that configure should have an option that accepts a path to a suitable m4 binary. If that option is passed, then configure sets a variable that gets put into config.h which main.c reads. If that option is not passed, then configure should try to find a suitable m4 and set the symbol as previously. The M4 environment variable is handy for certain situations that might arise as well.
> 
> Sounds good.
> 
> >
> > Thanks for your report and interest in flex.
> 
> You're welcome. And thank you for your quick response.
> 
> 
> ------------------------------------------------------------------------------
> Mobile security can be enabling, not merely restricting. Employees who
> bring their own devices (BYOD) to work are irked by the imposition of MDM
> restrictions. Mobile Device Manager Plus allows you to control only the
> apps on BYO-devices by containerizing them, leaving personal data untouched!
> https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
> -- 
> Flex-help mailing list
> Fle...@li...
> https://lists.sourceforge.net/lists/listinfo/flex-help

-- 
Will Estes
wes...@gm...

Re: [Flex-help] Some flex patch snippets for Solaris.

[Rich B. <ric...@or...>]

On 05/20/16 03:12 PM, Will Estes wrote:
> The bootstrapping of flex was reworked after 2.6.1 was released. I -think- it's immune to the vpath build that you do. If you can verify that against the current git tree, let me know the results. Ditto the building of the man page.

Confirmed. Both are now fine.

>
> As for finding m4, the solution needs to be more general. While what you do works for people who want to use the stock setup on Solaris, I can easily see other situations where it doesn't do enough.

Agreed.

>
> I'll open an issue to describe the problem.
>
> What should happen is that configure should have an option that accepts a path to a suitable m4 binary. If that option is passed, then configure sets a variable that gets put into config.h which main.c reads. If that option is not passed, then configure should try to find a suitable m4 and set the symbol as previously. The M4 environment variable is handy for certain situations that might arise as well.

Sounds good.

>
> Thanks for your report and interest in flex.

You're welcome. And thank you for your quick response.