flex-help Mailing List for flex: the fast lexical analyser

Hi all,

I'm trying to learn Bison and Flex so I can make a converter between a
markup language of my design and PDF or HTML, depending on which Bison
program I use. After having been unsuccessful following several online
docs, I've decided to make the simplest possible parser where my
trivial Flex program tokenizes a document file comprised of paragraphs
each separated by one blank line, and my trivial Bison program adds
"<p>" to the front of each paragraph, and </p> to the end of each
paragraph. On this trivial experiment, to make it easier, I can specify
that a blank line consists of two consecutive \n characters without any
space between them, and if necessary a \n at the beginning of the
document file and an extra \n at the end of the document file.

I can see two ways to do this:

1) All Flex to Bison communication happens via the scanner's stdout to
   the parser's stdin, using a distinct executable for the scanner and
   the parser respectively. Both the Flex program and the Bison program
   would each have a main() function.

2) All Flex to Bison communications happen through tokens defined in
   the Bison source file, yylval, $$ and $1 etc. The Flex program would
   be a library without its own main(), that gets compiled and linked
   into the Bison generated parser. The combined program then
   translates the input document coming in through its stdin into
   something else that comes out of its stdout

#1 is conceptually easy, but I haven't been able to do it. I got close,
   but I couldn't make Bison do the regex necessary to parse its stdin
   input.

#2 seems to be the "best practices" way and I'd imagine it's faster on
   big input files, but I haven't been able to make it work because I
   don't understand how it functions.

I know this text paragraphs to <p></p> surrounded paragraphs conversion
could easily be done in Flex only, or for that matter a five line AWK
program or even an AWK one liner. My purpose in making this thing is to
have the easiest possible program that actually passes tokens from the
scanner to the parser, which translates it into something else. It's
the "Hello World" I must slowly build from to create a real converter.

For #2 I'd imagine my Bison rules would look something like the

input: chunk | input chunks
chunk: paragraph newline


1) Has any of you communicated between your lexer and parser using
stdio/stdin or intermediate files exclusively?

2) For option 2, what would my Flex and Bison files look like, and what
   would be the command to compile and link  them together? If the
   executable for option #2 is a.out, would the command to use it be
   cat mytest.txt > ./a.out

Also, is there a Flex IRC channel?

I know it sounds like I'm asking you to do my homework, but I've been
trying to do this for two weeks with web research and experimentation,
and haven't gotten to first base, so I'd appreciate any help you could
give.

Thanks,

SteveT

Steve Litt 

Autumn 2023 featured book: Rapid Learning for the 21st Century
http://www.troubleshooters.com/rl21

Flex 2.6.4

I've looked at the Flex manual and I can't seem to find any way of 
inserting anything into the output header file. It would be useful to 
allow an option to put things into the header file.

Let me give an example. I have a bison parser using Flex. In the parser 
I detect an include statement (similar to a #include) which requires 
that activity in the current file be suspended, the data relevant to the 
current process be stacked, and a new one opened and processed. Flex 
provides a mechanism to do this. I would like to provide my own 
interface to the Flex functions to the Bison parser. There is no way to 
do this other than to hand code the function prototype. This hand coding 
is something that Flex can do if there were a means to insert a function 
prototype into the Flex generated header file.

Flex 2.6.4

In looking at the generated header file is looks like all the 
defines/undefs, except for YY_DECL, are local to and used only by the 
Flex generated source code. If that's the  case then is there any reason 
that they can not all be moved from the header file to the source file? 
That would ensure that the generated header file can be imported 
everywhere, including the flex generated source file. At the moment the 
header file can not be imported to the source file, and, except for 
YY_DECL, does not seem to contain anything useful for any other 
header/source file.

flex 2.6.4
Win 11-x64
g++ (GCC) 11.3.0
cygwin

 > g++ -c -std=c++11

The function isatty() in yy_init_buffer(), a flex generated function, is 
not compatible in C++ and generates an error. I am working on a 
work-around but the issue is that this requires that the generated file 
can not be used as is but must be edited. Is there a work-around for 
this? Is there an option which can select another function instead of 
isatty()?

The isatty() call is isatty( fileno( file ) ) where 'file' is of type 
FILE*. I don't know if this is also an issue.

I'd prefer not to do editing if that is possible.

thanks

Found the error. A missing rule closure, missing '}'. I found it by 
successively removing code/rules until the error disappeared. A 
decidedly crude approach.

This effort would have been made somewhat easier if the start line 
became part of the error message. In that way I could start my search at 
the point where the rule was being evaluated. Given the end line is 
deceptive since the end line, in this case, is the end of the program. 
Not knowing the rule being evaluated when the error was found, means 
that I have to begin this search at the start of the rules section 
('%%') and continue until the end of the rules section, (%%').

Is there any chance that a future version of the software could output 
the start line rather than the end line of all errors?

thanks

On 6/8/2023 9:58 AM, Arthur Schwarz wrote:
> Flex v2.6.4
> Bison v3.8.2
> Win 7/11-x64
>
> I am updating my Flex file from 2013  and integrating it with the 
> latest Bison. When I ran flex two 'unrecognized rule' errors were 
> generated. I've looked at the Flex 2.6.0 manual pg. 104 and 120 in the 
> FAQ to see what the error is and how to locate the point of error but 
> they don't appear to apply.
>
> ERASEME57,  pg. 104 says "flex can’t accommodate an action on a 
> separate line".
>
> unnamed-faq-90: pg. 120 says "You can’t use the ’/’ operator inside 
> parentheses".
>
> Neither of these two conditions seem to apply in my case. The error 
> message does provide a line number where the error was detected but 
> gives a line number at the end of rule processing, "slip.lex:464: 
> unrecognized rule".
>
> Is there any way of finding what 'rule' caused the error. I can't seem 
> to find what's wrong.
>
> My code begins on the same line as the rule, so ERASEME57 does not apply.
>
> The rules containing a '/' are not within a parenthesis, so 
> unnamed-faq-90 does not seem to apply. The rules are:
>     FOLLOW [,;:\(\)<>/{}|[:cntrl:]| ]
>     NAMEEND ([[:graph:]_\$]{-}[,;:\(\)<>/{}])
>     "/*"                             { DEBUG(lexerDebugFlag, 
> printf("Lex: %3d:%4d /* BEGIN COMMENT %s\n",    lineno, col, yytext);) 
> col++;
> BEGIN(COMMENT); }
>     "//"                             { DEBUG(lexerDebugFlag, 
> printf("Lex: %3d:%4d EOL COMMENT %s\n", lineno, col, yytext);) col++;
> BEGIN(EOLCOMMENT); }
>     .*"*/"                           { DEBUG(lexerDebugFlag, 
> printf("Lex: %3d:%4d END COMMENT `%s`\n", lineno, col, yytext);)
> BEGIN(INITIAL); }
>
> I'm stuck. Are there other issues associated with the "unrecognized 
> rule" message?
>
> Thanks
>

Flex v2.6.4
Bison v3.8.2
Win 7/11-x64

I am updating my Flex file from 2013  and integrating it with the latest 
Bison. When I ran flex two 'unrecognized rule' errors were generated. 
I've looked at the Flex 2.6.0 manual pg. 104 and 120 in the FAQ to see 
what the error is and how to locate the point of error but they don't 
appear to apply.

ERASEME57,  pg. 104 says "flex can’t accommodate an action on a separate 
line".

unnamed-faq-90: pg. 120 says "You can’t use the ’/’ operator inside 
parentheses".

Neither of these two conditions seem to apply in my case. The error 
message does provide a line number where the error was detected but 
gives a line number at the end of rule processing, "slip.lex:464: 
unrecognized rule".

Is there any way of finding what 'rule' caused the error. I can't seem 
to find what's wrong.

My code begins on the same line as the rule, so ERASEME57 does not apply.

The rules containing a '/' are not within a parenthesis, so 
unnamed-faq-90 does not seem to apply. The rules are:
     FOLLOW [,;:\(\)<>/{}|[:cntrl:]| ]
     NAMEEND ([[:graph:]_\$]{-}[,;:\(\)<>/{}])
     "/*"                             { DEBUG(lexerDebugFlag, 
printf("Lex: %3d:%4d /* BEGIN COMMENT %s\n",    lineno, col, yytext);) 
col++;
BEGIN(COMMENT); }
     "//"                             { DEBUG(lexerDebugFlag, 
printf("Lex: %3d:%4d EOL COMMENT %s\n", lineno, col, yytext);) col++;
BEGIN(EOLCOMMENT); }
     .*"*/"                           { DEBUG(lexerDebugFlag, 
printf("Lex: %3d:%4d END COMMENT `%s`\n", lineno, col, yytext);)
BEGIN(INITIAL); }

I'm stuck. Are there other issues associated with the "unrecognized 
rule" message?

Thanks

Am 07.03.2022 um 00:11 schrieb Will Estes:
> Do you need to modify flex itself? If not, you're better served by using one of the source distributions that will have  the intermediate files included so you can take up immediately with using the configure script which supports a number of arguments to specify where various things should end up. See https://github.com/westes/flex/releases/tag/v2.6.4 for the most current release of flex.

Sadly the last flex release was from early 2017, nearly 5 years ago.
That provides a bit of "pressure" to build flex from source.

When it is only about the intermediate files, then something like an 
automated build would help a lot, and this build can also run the tests 
on commits and pull requests.

@Will: Until one of those is in case - maybe you can publish a 2.6.5-dev 
snaphot that could be updated later by running "make dist" on your 
machine, so people would have a 2022 version to use if the 2017 release 
is "not enough"?

Thank you for considering that!
Simon

> 
> On Sunday,  6 March 2022,  9:00 pm +0000, Ashley Rohde <a.t...@ho...> wrote:
> 
>> Hello,
>>
>> I'm having trouble with the last installation step of flex (so close). The error messages are included below. I suspect the problem is that the autogen script tries to install something (libtoolize? autoreconf?) into the user directory, but I am working on a cluster computer that requires me to use a project file (/project/bee-genomics/ashley.rohde), which is where all of the supporting programs are installed. I can't see a place in the script to change this destination and I don't know what to do. I am pretty new to working from a command line and this is the first time I've had to compile a program. Any help anyone could provide is greatly appreciated.
>>
>> Thanks very much.
>>
>> Cheers,
>>
>> Ashley Rohde
>>
>> Error message:
>>
>> make[2]: Entering directory `/project/bee-genomics/ashley.rohde/autoconf-2.71'
>> make[2]: Warning: File `Makefile' has modification time 47 s in the future
>>   /usr/bin/mkdir -p '/user/share/autoconf/bin'
>> /usr/bin/mkdir: cannot create directory '/user': Permission denied
>> make[2]: *** [install-binSCRIPTS] Error 1
>> make[2]: Leaving directory `/project/bee-genomics/ashley.rohde/autoconf-2.71'
>> make[1]: *** [install-am] Error 2
>> make[1]: Leaving directory `/project/bee-genomics/ashley.rohde/autoconf-2.71'
>> make: *** [install] Error 2
>>
>>
>> --
>> Flex-help mailing list
>> Fle...@li...
>> https://lists.sourceforge.net/lists/listinfo/flex-help
> 
> --
> Will Estes
> wes...@gm...
> 
> 

Do you need to modify flex itself? If not, you're better served by using one of the source distributions that will have  the intermediate files included so you can take up immediately with using the configure script which supports a number of arguments to specify where various things should end up. See https://github.com/westes/flex/releases/tag/v2.6.4 for the most current release of flex.

On Sunday,  6 March 2022,  9:00 pm +0000, Ashley Rohde <a.t...@ho...> wrote:

> Hello,
>
> I'm having trouble with the last installation step of flex (so close). The error messages are included below. I suspect the problem is that the autogen script tries to install something (libtoolize? autoreconf?) into the user directory, but I am working on a cluster computer that requires me to use a project file (/project/bee-genomics/ashley.rohde), which is where all of the supporting programs are installed. I can't see a place in the script to change this destination and I don't know what to do. I am pretty new to working from a command line and this is the first time I've had to compile a program. Any help anyone could provide is greatly appreciated.
>
> Thanks very much.
>
> Cheers,
>
> Ashley Rohde
>
> Error message:
>
> make[2]: Entering directory `/project/bee-genomics/ashley.rohde/autoconf-2.71'
> make[2]: Warning: File `Makefile' has modification time 47 s in the future
>  /usr/bin/mkdir -p '/user/share/autoconf/bin'
> /usr/bin/mkdir: cannot create directory '/user': Permission denied
> make[2]: *** [install-binSCRIPTS] Error 1
> make[2]: Leaving directory `/project/bee-genomics/ashley.rohde/autoconf-2.71'
> make[1]: *** [install-am] Error 2
> make[1]: Leaving directory `/project/bee-genomics/ashley.rohde/autoconf-2.71'
> make: *** [install] Error 2
>
>
> --
> Flex-help mailing list
> Fle...@li...
> https://lists.sourceforge.net/lists/listinfo/flex-help

--
Will Estes
wes...@gm...

Hello,

I'm having trouble with the last installation step of flex (so close). The error messages are included below. I suspect the problem is that the autogen script tries to install something (libtoolize? autoreconf?) into the user directory, but I am working on a cluster computer that requires me to use a project file (/project/bee-genomics/ashley.rohde), which is where all of the supporting programs are installed. I can't see a place in the script to change this destination and I don't know what to do. I am pretty new to working from a command line and this is the first time I've had to compile a program. Any help anyone could provide is greatly appreciated.

Thanks very much.

Cheers,

Ashley Rohde

Error message:

make[2]: Entering directory `/project/bee-genomics/ashley.rohde/autoconf-2.71'
make[2]: Warning: File `Makefile' has modification time 47 s in the future
 /usr/bin/mkdir -p '/user/share/autoconf/bin'
/usr/bin/mkdir: cannot create directory '/user': Permission denied
make[2]: *** [install-binSCRIPTS] Error 1
make[2]: Leaving directory `/project/bee-genomics/ashley.rohde/autoconf-2.71'
make[1]: *** [install-am] Error 2
make[1]: Leaving directory `/project/bee-genomics/ashley.rohde/autoconf-2.71'
make: *** [install] Error 2

Hi, I'm a newbee to Flex, I wrote a very simple test.l file to trace how the flex run.
The test.l is shown as follow:...   %%         [aeiou]+   {printf("aeiou\n");}         ^r             {printf("^r\n");}   %%...
I used flex-2.6.0 to create the lex.yy.c from it. I traced the source code, everything went fine until the ntod function(in dfa.c). It mainly transforms the NFA to DFA state. I know how the epsilon-closure is created. Which confused me is that:
1. what is the sympartition function used for ? (Accord to my understanding, the .l specification created three equivalance class: aeiou、r、and . for any other else. the ccltbl has converted to their equivalance class number. so why does it use the mkeccl function to get the duplist, and what is the duplist used for ? and the cclng ? Could you explain the detail of sympartition ?)
2. why the duplist[sym] == NIL means symbol has unique out-transitions ? how does the symfollowset work ?
The NFA state is shown as below:
********** beginning dump of nfa with start state 11
state #    1    257:     0,    0
state #    2    257:     0,    0
state #    3     -1:     4,    0
state #    4    257:     3,    0  [1]
state #    5    257:     1,    3
state #    6    114:     7,    0
state #    7    257:     0,    0  [2]
state #    8    257:     2,    6
state #    9     -2:    10,    0
state #   10    257:     0,    0  [3]
state #   11    257:     5,    9
********** end of dump

Whoa, sweet! One of the authors of the book I use as a reference answered
me!

Thank you for your answers! Would you mind going a bit deeper?

For the "push-lexer" use case, I am thinking about the microcontroller
world, where I don't have threads (although it IS a 32bit MCU). Or do you
recommend other solutions (i.e. not lex&yacc) for online embedded protocol
parsing?
Right now I am scanning for end of line and I push complete lines with
yy_scan_bytes, but it feels silly.
I'll give a try to implementing a "come back later, I might have new data"
function. Do you reckon it is something that might be merged?


For the second question does "the state is in the grammar, not the
tokenizer" mean that the tokenizer does not tokenize any more and passes
only string that are analyzed in the parser or does it work with start
states? And in the case of start states, is there a recommended way of
doing it?

Cheers,
Jérôme.


On Sat, Jan 23, 2021 at 7:59 PM John Levine <jo...@ta...> wrote:

> In article <
> CAE...@ma...> you
> write:
> >I would like to build a push parser, and would like the lexer to behave in
> >accordance.
>
> While that is a reasonable thing to do, flex doesn't do that. When I
> wrote flex&bison in 2009 I noted that bison can create push parsers
> but flex can't.  As far as I can see, nothing has changed there.
>
> While it would be possible in principle to modify flex to do push scanning,
> it'd be a lot of work.  Every environment supports threads these days so
> I would fake it by running the scanner and parser in its own thread.
>
> >Also, on another point, I sometimes feel that I need to enter a new start
> >state but this is not recognizable by the lexer, but only by the parser.
> Am
> >I doing something wrong? Is the grammar I am trying to implement broken?
> Or
> >is it legitimate?
>
> That is quite common. There's lots of situations where the information
> about the state is in the grammar, not the tokenizer.
>
> R's,
> John
>

In article <CAE...@ma...> you write:
>I would like to build a push parser, and would like the lexer to behave in
>accordance.

While that is a reasonable thing to do, flex doesn't do that. When I
wrote flex&bison in 2009 I noted that bison can create push parsers
but flex can't.  As far as I can see, nothing has changed there.

While it would be possible in principle to modify flex to do push scanning,
it'd be a lot of work.  Every environment supports threads these days so
I would fake it by running the scanner and parser in its own thread.

>Also, on another point, I sometimes feel that I need to enter a new start
>state but this is not recognizable by the lexer, but only by the parser. Am
>I doing something wrong? Is the grammar I am trying to implement broken? Or
>is it legitimate?

That is quite common. There's lots of situations where the information
about the state is in the grammar, not the tokenizer.

R's,
John

Hi,

Thank you all for your excellent work on flex, I am happy to be able to
stand on the shoulders of giants (not that I compare myself the least to
the genius Newton).

Now, I am really sorry if I missed a ressource that documents my needs, and
I would be grateful to be pointed to it.

I would like to build a push parser, and would like the lexer to behave in
accordance.
I found this excellent ressource (https://github.com/mwberry/yacc-examples),
but its data is available beforehand or needs to be fed through blocking in
YY_INPUT (for example with a thread, which I cannot use in my case). This
is what I would call a PULL lexer.

Is it (or would it be without breaking the internals of flex) possible to
return a special value from YY_INPUT saying "I don't have any more data
JUST YET" and later being able to be called again and feed the new data
(either from YY_INPUT or from a buffer re-filling function). This would
then be a PUSH lexer.

Also, on another point, I sometimes feel that I need to enter a new start
state but this is not recognizable by the lexer, but only by the parser. Am
I doing something wrong? Is the grammar I am trying to implement broken? Or
is it legitimate?

Best regards,
Jérôme.

Greetings to all and thanks advances for paying attention to my mail


I have a mistake just baffling

I run muy first program and next gets output correct

Number : 1
.
Number : 1
.
Number : 0
-
Stage : snapshot
Build : 20190729201751



So y run a second program but i get a incorrect result
Name :1

Should be Number : 1 like in first ran program


The program entry is the same, they differ in that one is the test
program (cunit) and the second is the productive program, in both cases
the complete scanner is called from a shared library generated in cmake
and the scanner is generated in C ++

the firts rule in flex file is 

{DIGIT}{1,11}   {
                                        std::cout << "Number : " <<
yytext << std::endl; 
                                        yylval->build< int >(
atoi(yytext) );
                                        return token::NUMBER;
                                }


whe run the second program no call this rule it call final rule

{NAME}          {std::cout << "Name: " << yytext << std::endl; yylval-
>build< std::string >( yytext ); return token::NAME;}


my cuestion is why this differ is the same input, the same class scaner
only diferent program.

I attach my flex file, if you need the rest I put a link for both

firt program https://github.com/azaeldevel/toolkit.git commit
8b4845dda686321fbaf0d9e6a5abc04992dd6f24
second program https://github.com/azaeldevel/apidb.git commit
c887addfdc0a7fbad80df3571a8214c5e0bb8d34


I appreciate any comments

Hi,

I'm trying to help a friend set up autoconf on their project which uses 
flex. I've encountered a kind of a funny situation which is probably 
quite easy to handle, but I don't see the right way to handle it. It has 
to do with his definition of YY_INPUT, which looks like this:

#define YY_INPUT(buf,result,max_size) fussy_inp(buf,&result,max_size)

Later on, he has the function definition:

void fussy_inp(char *buf, int *result, int max_size) { ...

Under flex 2.5, this is the right prototype, but under 2.6 it seems to 
want yy_size_t* instead.

I feel like the right thing to do might be to make it a macro instead of 
a function, but it does some significant computation and has a static 
variable in it, so if I could leave it as a function I'd be happier.

At first I thought I'd use AC_CHECK_TYPE to see if yy_size_t exists but 
I think it does exist in 2.5. My next guess was to try checking the type 
of a struct member, but this is getting to be a hairy amount of code in 
autoconf and I thought I'd check here and see if I'm missing something 
obvious first.

https://stackoverflow.com/questions/52296527/autoconf-check-struct-member-type

Thanks,

-- 
Daniel Lyons

Hi and welcome to the world of lexers!

Am 27.02.2019 um 14:36 schrieb Wor...@gm...:
5. .|\n                    { printf("do nothing"); }
6. [_a-zA-Z0-9]+\(     { printf("This is a function name %s
   (steps=%i)", yytext, steps++); }

>> Now my problem is that also strings like 12342welcome( are beeing
>> "marked" as beeing functions ... what am i doing wrong here ? the
>> default route .|\n is said to do nothing, so why does he matches also
>> digits with my function catching example ?

Actually you've said that anything that consist only of *at least one*
("+") from the following set "[_a-zA-Z0-9]" (directly followed by a "(")
is a function.
As you did not specify what you want to do you may want to mark only
functions if they not start with a number, using a fixed and an optional
set (* matches zero to unlimited occurrences):

[_a-zA-Z]+[_a-zA-Z0-9]*\(

you may also want to add optional spaces before "\(".

> I've replaced the lines 5 and 6 because he matches first the lines
> with the most matches and that is normaly .|\n but again same problem
> =(.

If I'm not wrong the *longest* match is the preferred one and as this
rule matches only a single character it should normally not be used.

Simon

I've replaced the lines 5 and 6 because he matches first the lines with 
the most matches and that is normaly .|\n but again same problem =(.

On 27.02.19 14:00, Wor...@gm... wrote:
> Hi everyone, i've got me the book flex&bison and have troubles with 
> one of the first examples:
>
>
> 1. %{
>
> 2. #include <stdio.h>
>
> 3. %}
>
> 4. %%
>
> 5. [_a-zA-Z0-9]+\(     { printf("This is a function name %s 
> (steps=%i)", yytext, steps++); }
> 6. .|\n                    { printf("do nothing"); }
>
> 7. %%
> 8. int main(int argc, char **argv)
> 9.{
> 10.    yylex();
> 11.    return(0);
> 12. }
>
> Now my problem is that also strings like 12342welcome( are beeing 
> "marked" as beeing functions ... what am i doing wrong here ? the 
> default route .|\n is said to do nothing, so why does he matches also 
> digits with my function catching example ?
>
>
> best regards!
>
>
>

Hi everyone, i've got me the book flex&bison and have troubles with one 
of the first examples:


%{

#include <stdio.h>

%}

%%

[_a-zA-Z0-9]+\(     { printf("This is a function name %s (steps=%i)", 
yytext, steps++); }
.|\n                    { printf("do nothing"); }

%%

int main(int argc, char **argv)
{

     yylex();

     return(0);

}

Now my problem is that also strings like 12342welcome( are beeing 
"marked" as beeing functions ... what am i doing wrong here ? the 
default route .|\n is said to do nothing, so why does he matches also 
digits with my function catching example ?


best regards!

Wrong version posted.
It is 2.6.1

On Wed 6 Jun, 2018, 3:13 PM Will Estes, <wes...@gm...> wrote:

> Your parser is generated by bison, not flex. "3.0.5" is a bison version
> number, not a flex version number.
>
> On Wednesday,  6 June 2018,  8:14 am +0530, Anand Akhare <
> ana...@gm...> wrote:
>
> > In below code in generated parser, yy_init and yy_start is showing
> > uninitialized value. irrelevant crash (at while ( /* CONSTCOND */ 1) is
> > happening and below code doesn't get executed. My flex is now 3.0.5
> >
> > > Can this be problem? Right now, after yy.c is generated, I am putting
> > > these values zero above this code. Similarly for yy_buffer_stack and
> > > buffer_stack_max.
> > >  Scanner is with reentrant option and prefix option.
> > > With prefix option, yyin is not getting prefix, is it true?
> > > yyin can conflict from first instance of scanner with second instance
> of
> > > scanner.
> > >
> > > This
> > >         if ( !yyg->yy_init )
> > >                 {
> > >                 yyg->yy_init = 1;
> > >
> > > #ifdef YY_USER_INIT
> > >                 YY_USER_INIT;
> > > #endif
> > >
> > >                 if ( ! yyg->yy_start )
> > >                         yyg->yy_start = 1;      /* first start state */
> > >
> > >                 if ( ! yyin )
> > >                         yyin = stdin;
> > >
> > > >>>>               yyg->yy_buffer_stack = NULL;
> > > >>>>                yyg->yy_buffer_stack_max = 0;
> > >                 if ( ! YY_CURRENT_BUFFER ) {
> > >                         prepyyensure_buffer_stack (yyscanner);
> > >                         YY_CURRENT_BUFFER_LVALUE =
> > >                                 prepyy_create_buffer(yyin,YY_BUF_SIZE
> > > ,yyscanner);
> > >                 }
> > >
> > >                 prepyy_load_buffer_state(yyscanner );
> > >
> > > Many Thanks,
> > > Anand
> > >
> > >
> >
> ------------------------------------------------------------------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> > --
> > Flex-help mailing list
> > Fle...@li...
> > https://lists.sourceforge.net/lists/listinfo/flex-help
>
> --
> Will Estes
> Flex Project Maintainer
> wes...@gm...
> https://github.com/westes/flex
>

Your parser is generated by bison, not flex. "3.0.5" is a bison version number, not a flex version number.

On Wednesday,  6 June 2018,  8:14 am +0530, Anand Akhare <ana...@gm...> wrote:

> In below code in generated parser, yy_init and yy_start is showing
> uninitialized value. irrelevant crash (at while ( /* CONSTCOND */ 1) is
> happening and below code doesn't get executed. My flex is now 3.0.5
> 
> > Can this be problem? Right now, after yy.c is generated, I am putting
> > these values zero above this code. Similarly for yy_buffer_stack and
> > buffer_stack_max.
> >  Scanner is with reentrant option and prefix option.
> > With prefix option, yyin is not getting prefix, is it true?
> > yyin can conflict from first instance of scanner with second instance of
> > scanner.
> >
> > This
> >         if ( !yyg->yy_init )
> >                 {
> >                 yyg->yy_init = 1;
> >
> > #ifdef YY_USER_INIT
> >                 YY_USER_INIT;
> > #endif
> >
> >                 if ( ! yyg->yy_start )
> >                         yyg->yy_start = 1;      /* first start state */
> >
> >                 if ( ! yyin )
> >                         yyin = stdin;
> >
> > >>>>               yyg->yy_buffer_stack = NULL;
> > >>>>                yyg->yy_buffer_stack_max = 0;
> >                 if ( ! YY_CURRENT_BUFFER ) {
> >                         prepyyensure_buffer_stack (yyscanner);
> >                         YY_CURRENT_BUFFER_LVALUE =
> >                                 prepyy_create_buffer(yyin,YY_BUF_SIZE
> > ,yyscanner);
> >                 }
> >
> >                 prepyy_load_buffer_state(yyscanner );
> >
> > Many Thanks,
> > Anand
> >
> >
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> -- 
> Flex-help mailing list
> Fle...@li...
> https://lists.sourceforge.net/lists/listinfo/flex-help

-- 
Will Estes
Flex Project Maintainer
wes...@gm...
https://github.com/westes/flex

In below code in generated parser, yy_init and yy_start is showing
uninitialized value. irrelevant crash (at while ( /* CONSTCOND */ 1) is
happening and below code doesn't get executed. My flex is now 3.0.5

> Can this be problem? Right now, after yy.c is generated, I am putting
> these values zero above this code. Similarly for yy_buffer_stack and
> buffer_stack_max.
>  Scanner is with reentrant option and prefix option.
> With prefix option, yyin is not getting prefix, is it true?
> yyin can conflict from first instance of scanner with second instance of
> scanner.
>
> This
>         if ( !yyg->yy_init )
>                 {
>                 yyg->yy_init = 1;
>
> #ifdef YY_USER_INIT
>                 YY_USER_INIT;
> #endif
>
>                 if ( ! yyg->yy_start )
>                         yyg->yy_start = 1;      /* first start state */
>
>                 if ( ! yyin )
>                         yyin = stdin;
>
> >>>>               yyg->yy_buffer_stack = NULL;
> >>>>                yyg->yy_buffer_stack_max = 0;
>                 if ( ! YY_CURRENT_BUFFER ) {
>                         prepyyensure_buffer_stack (yyscanner);
>                         YY_CURRENT_BUFFER_LVALUE =
>                                 prepyy_create_buffer(yyin,YY_BUF_SIZE
> ,yyscanner);
>                 }
>
>                 prepyy_load_buffer_state(yyscanner );
>
> Many Thanks,
> Anand
>
>

Am 16.03.2018 um 22:05 schrieb Will Estes:
> I don't think the one open pull request against 2.6.5 will effect> the performance you're seeing, so just see how your scanner does
> against whatever the tip of master happens to be.

I did a new check today (which resulted in an issue and some minor PR
;-) - and the result is a mixed one.

The good thing first: the CPU cycles spent in the scanner got down about
18-23% (depends on the actual run as the machine is not idlying).

But the scanner still takes around 20% of the complete parsing process
and the compiler consists of much more than the plain scanner...

this is what callgrind reports in scanner.c:

Complete part of run: 42%, 5.8% in yy_get_next_buffer, 14.6% in yylex

I assume this isn't a general bug in flex but a weakness in the scanner:
https://sourceforge.net/p/open-cobol/code/HEAD/tree/trunk/cobc/scanner.l

Can anybody please have a look and drop some hints?

Thank you very much,
Simon

I don't think the one open pull request against 2.6.5 will effect the performance you're seeing, so just see how your scanner does against whatever the tip of master happens to be.

On Friday, 16 March 2018, 10:01 pm +0100, Simon Sobisch <sim...@gn...> wrote:

> Thank you for answering and about the notes about the progress.
> 
> I'll try 2.6.5 dev on Monday or Tuesday (or whenever the pull you
> refered to was done) - Can you please leave a link to the PR allowing me
> to subscribe to it?
> I'll report back next week when I have access to the test environment.
> 
> If someone stumbles over bad practice in the referred scanner.l I'd
> still like to know about it ;-)
> 
> Thank you for the analyzer,
> Simon
> 
> Am 16.03.2018 um 21:32 schrieb Will Estes:
> > There have been some performance changes since that version, yes, so first I would see how things stand with 2.6.4. There will be a 2.6.5 soonish -- one more pull request remains and then I'll be putting in tooling in to make releasing flex easier -- just to give you a sense of the timeline for 2.6.5 which also contains a fair number of changes since 2.6.4.
> > 
> > On Friday, 16 March 2018,  9:05 pm +0100, Simon Sobisch <sim...@gn...> wrote:
> > 
> >>
> >> Hi flex-users and developers,
> >>  
> >> I've just did a profiling (using callgrind) on the GnuCOBOL compiler
> >> (parsing only) with 2,500,000 LOC and found out that the recent
> >> scanner[1] consumes most of the time:
> >>
> >> * 44% spent in yylex (16% of these are spent in GnuCOBOL internal
> >> functions leaving 28% of the complete parser process in this single flex
> >> generated file)
> >>
> >> * 13% in yy_get_previous_state
> >>
> >> * 5.6% spent in yy_get_next_buffer with 1.3% in fgets
> >>  
> >> Removing all the "noise" there is still over 25% spent in the scanner
> >> and "get_previous_state" doesn't sound like it should be called that often.
> >>  
> >> I guess there are some bad practices used in scanner.l, but I don't know
> >> what to look out for.
> >> Can you please have a look at this and post suggestions or point to bad
> >> practice found in there?
> >>  
> >> Is the amount of calls to get_previous_state normal?
> >>  
> >> scanner.c was generated with flex 2.5.37 - Do you expect a speedup with
> >> a more recent release?
> >>  
> >> Thank you for taking the time to answer and maybe even for checking the
> >> source,
> >> Simon
> >>
> >> [1] https://sourceforge.net/p/open-cobol/code/HEAD/tree/trunk/cobc/scanner.l
> >>
> >> ------------------------------------------------------------------------------
> >> Check out the vibrant tech community on one of the world's most
> >> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> >> -- 
> >> Flex-help mailing list
> >> Fle...@li...
> >> https://lists.sourceforge.net/lists/listinfo/flex-help
> > 

-- 
Will Estes
wes...@gm...

Thank you for answering and about the notes about the progress.

I'll try 2.6.5 dev on Monday or Tuesday (or whenever the pull you
refered to was done) - Can you please leave a link to the PR allowing me
to subscribe to it?
I'll report back next week when I have access to the test environment.

If someone stumbles over bad practice in the referred scanner.l I'd
still like to know about it ;-)

Thank you for the analyzer,
Simon

Am 16.03.2018 um 21:32 schrieb Will Estes:
> There have been some performance changes since that version, yes, so first I would see how things stand with 2.6.4. There will be a 2.6.5 soonish -- one more pull request remains and then I'll be putting in tooling in to make releasing flex easier -- just to give you a sense of the timeline for 2.6.5 which also contains a fair number of changes since 2.6.4.
> 
> On Friday, 16 March 2018,  9:05 pm +0100, Simon Sobisch <sim...@gn...> wrote:
> 
>>
>> Hi flex-users and developers,
>>  
>> I've just did a profiling (using callgrind) on the GnuCOBOL compiler
>> (parsing only) with 2,500,000 LOC and found out that the recent
>> scanner[1] consumes most of the time:
>>
>> * 44% spent in yylex (16% of these are spent in GnuCOBOL internal
>> functions leaving 28% of the complete parser process in this single flex
>> generated file)
>>
>> * 13% in yy_get_previous_state
>>
>> * 5.6% spent in yy_get_next_buffer with 1.3% in fgets
>>  
>> Removing all the "noise" there is still over 25% spent in the scanner
>> and "get_previous_state" doesn't sound like it should be called that often.
>>  
>> I guess there are some bad practices used in scanner.l, but I don't know
>> what to look out for.
>> Can you please have a look at this and post suggestions or point to bad
>> practice found in there?
>>  
>> Is the amount of calls to get_previous_state normal?
>>  
>> scanner.c was generated with flex 2.5.37 - Do you expect a speedup with
>> a more recent release?
>>  
>> Thank you for taking the time to answer and maybe even for checking the
>> source,
>> Simon
>>
>> [1] https://sourceforge.net/p/open-cobol/code/HEAD/tree/trunk/cobc/scanner.l
>>
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> -- 
>> Flex-help mailing list
>> Fle...@li...
>> https://lists.sourceforge.net/lists/listinfo/flex-help
> 

There have been some performance changes since that version, yes, so first I would see how things stand with 2.6.4. There will be a 2.6.5 soonish -- one more pull request remains and then I'll be putting in tooling in to make releasing flex easier -- just to give you a sense of the timeline for 2.6.5 which also contains a fair number of changes since 2.6.4.

On Friday, 16 March 2018,  9:05 pm +0100, Simon Sobisch <sim...@gn...> wrote:

> 
> Hi flex-users and developers,
>  
> I've just did a profiling (using callgrind) on the GnuCOBOL compiler
> (parsing only) with 2,500,000 LOC and found out that the recent
> scanner[1] consumes most of the time:
> 
> * 44% spent in yylex (16% of these are spent in GnuCOBOL internal
> functions leaving 28% of the complete parser process in this single flex
> generated file)
> 
> * 13% in yy_get_previous_state
> 
> * 5.6% spent in yy_get_next_buffer with 1.3% in fgets
>  
> Removing all the "noise" there is still over 25% spent in the scanner
> and "get_previous_state" doesn't sound like it should be called that often.
>  
> I guess there are some bad practices used in scanner.l, but I don't know
> what to look out for.
> Can you please have a look at this and post suggestions or point to bad
> practice found in there?
>  
> Is the amount of calls to get_previous_state normal?
>  
> scanner.c was generated with flex 2.5.37 - Do you expect a speedup with
> a more recent release?
>  
> Thank you for taking the time to answer and maybe even for checking the
> source,
> Simon
> 
> [1] https://sourceforge.net/p/open-cobol/code/HEAD/tree/trunk/cobc/scanner.l
> 
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> -- 
> Flex-help mailing list
> Fle...@li...
> https://lists.sourceforge.net/lists/listinfo/flex-help

-- 
Will Estes
Flex Project Maintainer
wes...@gm...
https://github.com/westes/flex