#91 Flex 2.5.35 fails on .l-file where 2.5.4 succeeds

2.6.0
open
nobody
5
2015-11-18
2008-11-05
Anonymous
No

From: Olaf Seibert <olafs@cs.ru.nl>

Flex version 2.5.35 has a smaller capacity than version 2.5.4, or it is less efficient in processing input rules.

I have a flex input file that is handled fine by version 2.5.4 (which is installed in the FreeBSD base system) but suddenly failed mysteriously when I upgraded my FreeBSD ports, which installed the later and more limited flex.

The vesion from ports fails:

$ /usr/local/bin/flex -pv -i -t morph.l > morph.c
flex: input rules are too complicated (>= 32000 NFA states)
flex version 2.5.35 usage statistics:
scanner options: -iptvI8 -Cem
31000/32000 NFA states
0/1000 DFA states (0 words)
949 rules
No backing up
7/40 start conditions
16901 epsilon states, 8918 double epsilon states
no character classes
0 state/nextstate pairs created
0/0 unique/duplicate transitions
0/1000 base-def entries created
0/2000 (peak 0) nxt-chk entries created
0/2500 (peak 0) template nxt-chk entries created
0 empty table entries
0 protos created
0 templates created, 0 uses
0/256 equivalence classes created
0/256 meta-equivalence classes created
0 (0 saved) hash collisions, 0 DFAs equal
38 sets of reallocations needed
256 total table entries needed

The older version from the base system works:

$ /usr/bin/flex -pv -i -t morph.l > morph.c
/usr/bin/flex version 2.5.4 usage statistics:
scanner options: -iptvI8 -Cem
28641/29000 NFA states
21996/23000 DFA states (5032166 words)
1567 rules
Compressed tables always back-up
Beginning-of-line patterns used
7/40 start conditions
8886 epsilon states, 4488 double epsilon states
22/100 character classes needed 157/500 words of storage, 145 reused
614918 state/nextstate pairs created
510844/104074 unique/duplicate transitions
22017/23000 base-def entries created
73510/74000 (peak 73731) nxt-chk entries created
84/2500 (peak 819) template nxt-chk entries created
3 empty table entries
750 protos created
21 templates created, 31 uses
39/256 equivalence classes created
4/256 meta-equivalence classes created
98 (55 saved) hash collisions, 488863 DFAs equal
110 sets of reallocations needed
191349 total table entries needed

Discussion

  • Joesph Krahn

    Joesph Krahn - 2008-11-05

    I just discovered that flex 2.5.35 can erroneously detect the use of REJECT, if REJECT appears in a substring, or in a C++ style (//...) comment. If you don't use REJECT, try adding "%option noreject" and see if this fixes it. You might also try "%option noyymore". It's possible that auto-detection of yymore is also broken.

     
  • Nobody/Anonymous

    Thanks for looking at this. I do indeed not use REJECT or yymore, but unfortunately adding those options doesn't help.
    I have temporarily placed the morph.l file here: http://twoquid.cs.ru.nl/phasar/morph.l so you can have a better look at it.
    Thanks in advance!

     
  • Will Estes

    Will Estes - 2008-12-28
    • milestone: --> 2.5.35
    • assigned_to: nobody --> wlestes
     
  • Will Estes

    Will Estes - 2012-02-04
    • assigned_to: wlestes --> nobody
     
  • Will Estes

    Will Estes - 2015-11-18
    • Group: 2.5.35 --> 2.6.0
     

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks