flex-devel — development of flex itself

[Flex-devel] yytext and similar in -P scanners

[Petr M. <pma...@re...>]

Hi list,

the editor `vile' (aka "vi like emacs") currently fails to build with
new flex, i.e. flex post 2.5.4a.  The problem is that in scanners
generated with -P, symbols yytext, yyleng, yylineno and others are not
available.  Documentation states that they should be:

> Within your scanner itself, you can still refer to the global
> variables and functions using either version of their name; but
> externally, they have the modified name.

I'm sending the patch that should fix this problem.  It does the same
thing old flex does, namely it introduces #defines for yy symbols.

Comments welcome.

Thanks,
PM

Re: [Flex-devel] Flex extension

[Will E. <wl...@us...>]

Sorry to take so long to get back to you about this. I know you've been
interested in this for a while.

At this point, I'm going to have to put this on hold as we're gearing up
for some flex releases over the next few months and we've got to work
out a lot of bugs, issues with documentation and a few small feature
requests, along with trying to get flex stable.

I think what you've done is worth looking at later, but I can't tell you
now whether or not it's a good idea to include it into flex.

Thanks,

--Will


On Saturday, 25 November 2006,14:53 +0100, Christian Judt wrote:

> Hello,
> 
> This summer I have completed my master thesis about how to manage
> named elements in regular expressions in the context of scanner generators.
> 
> During this work and after its completion I created an extension of
> Flex 2.5.33 which implements such a system.
> 
> Currently it is something like a beta version, but it may be of practical
> interest.
> 
> It is located at:
> http://members.aon.at/judt/christian/FlexExtensionEnglish.html
> 
> In this context I am now faced with a Question:
> Could that extension be worthy enough for further development?
> Or is it more of theoretical interest?
> 
> Any suggestions?
> 
> - Christian
> 
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Flex-devel mailing list
> Fle...@li...
> https://lists.sourceforge.net/lists/listinfo/flex-devel

[Flex-devel] Flex extension

[Christian J. <chr...@ao...>]

Hello,

This summer I have completed my master thesis about how to manage
named elements in regular expressions in the context of scanner generators.

During this work and after its completion I created an extension of
Flex 2.5.33 which implements such a system.

Currently it is something like a beta version, but it may be of practical
interest.

It is located at:
http://members.aon.at/judt/christian/FlexExtensionEnglish.html

In this context I am now faced with a Question:
Could that extension be worthy enough for further development?
Or is it more of theoretical interest?

Any suggestions?

- Christian

[Flex-devel] posting patches

[Will E. <wl...@us...>]

All,

It recently came to my attention that I wasn't subscribed to the
flex-devel mailing list. As maintainer, it's definitely a good thing
that I be in touch with folks who are interested in the guts of the flex
program.

There was a question posted back in May regarding how best to propose
patches for inclusion in flex.

In general there's not a simple answer, but there are some basic
guidelines that it'd be helpful to keep in mind:

* Shorter patches are easy to evaluate.

* Too many patches in too short a time can flood my capacity to evaluate
  them.

* If you have a set of related patches, keep them together.

* If you're proposing some big overriding change to flex, talk about it
  on flex-devel first.

As for the particular documentation patch in question, I just
incorporated (most of) it into the flex cvs tree.

So, thanks, Dustin for submitting the patch.

--Will

[Flex-devel] Prefixes in a reentrant scanner

[Dustin L. <du...@la...>]

I've started looking at flex with an eye towards building a new and
cleaner C++ interface based, perhaps, on the reentrant C scanner code.
(I'm happy to talk about this if anyone is interested.)

I'm studying the code but I'm not at all familiar with flex internals,
so I'm starting from scratch and appreciate all the help I can get.  My
first question is about the feasibility of a straightforward development
from the reentrant C code.

If I generate two reentrant C scanners, then I have to give each a
different prefix--say yy and zz.  Obviously yylex and zzlex have (in
general) different implementations, so the question is about the other
symbols that access the yyscanner argument.  Is this merely to
disambiguate the C namespace, where I cannot have two objects named,
say, yylex_init but one will be generated for each scanner, or does the
implementation of yylex_init and zzlex_init necessarily vary depending
on the chosen options and rules?

That probably isn't a very clear question, so let me try to elaborate.
The reentrant interface is already mostly object-oriented in principle,
and at first glance maps cleanly to a C++ class.  Take my first mockup,
the simplest lexer class that knows how to do anything:

namespace flex
{

class lexer
{
public:

    lexer() : my_lexer(NULL)
    {
        int failed = yylex_init(&my_lexer);
        assert(!failed); // eventually do something more intelligent here
    };

    virtual ~lexer()
    {
        yylex_destroy(my_lexer);
        my_lexer = NULL;
    };

    virtual int lex() { return yylex(my_lexer); };

private:

    yyscan_t my_lexer;
};

}

Notice how the interface looks appropriate: the class owns a yyscan_t
(and probably would eventually inherit from struct yyguts_t directly
rather than owning a pointer to one, but this is a mock-up) and the ctor
and dtor take care of init and destroy.  The next logical step would be
to make lex() pure virtual for overriding in a concrete subclass
containing the actual lexer function.  Except for one thing: you can't
write a generic base class, because the names would change in different
subclasses: they're supposed to call ${PREFIX}lex_init(&my_lexer), not a
generic lex_init that knows how to deal with any yyguts_t object.

So the question is really whether the reentrant lexer implementation is
such that it is feasible to generate generic lex_init, lex_destroy, and
so on functions, or whether something about the implementation requires
them to be separate functions.  If the former, then probably there is a
reasonably straightforward path ahead.  If the latter, then the job is
more difficult.  It seems that this would be a desirable approach for
the C scanner too, since it would allow more code to be shared between
scanners, but perhaps there is a good reason it isn't that way.

I've started poking around inside flex to try to find some answers, but
I can see that the learning curve is steep so I'm hoping someone can
keep me pointed in the right direction.  One thing I've learned is that
the vast majority of fields are identical between yyFlexLexer and
yyguts_t, so code sharing certainly should be possible (which ought to
make the maintainers happy).  I have a general idea of how I think it
should work.

Dustin

[Flex-devel] Documentation patch

[Dustin L. <du...@la...>]

Here is a small documentation patch for your consideration: it just adds
%option noyywrap to the reentrant scanner example so it builds as-is in
the simplest possible way (examples should be kind to n00bs) and also
adds a bit of whitespace to make it look a bit cleaner.  (The unindented
return in main() was especially blecherous.)

What would the maintainers prefer: small individual patches like this so
they're easy to evaluate individually, or a big patch with a bunch of
misc. fixes like this to reduce the number of patches?

Dustin

Re: [Flex-devel] Dumb question about building CVS flex

[Dustin L. <du...@la...>]

OK, after spending a bunch of time learning more about autotools than I
did before and learning how to get most of it configured, I finally
noticed that there *is* a special README for the cvs snapshot that
wasn't there whenever it was I last looked at flex sources.
./autogen.sh does all the pre-configure steps I was painfully
re-constructing by hand without the benefit of a clue.

*Bad programmer* *whack* *whack*

Dustin

[Flex-devel] Dumb question about building CVS flex

[Dustin L. <du...@la...>]

I'm trying to get up to speed with CVS flex, and am not sure if I've
done things wrong through sheer ignorance of autotools.  I checked out
flex like this:

    $ cvs -z3 -d:pserver:ano...@fl...:/cvsroot/flex co -P flex

and then ran 'aclocal' in the flex directory thus created.  This
produced some warnings:

---
/usr/share/aclocal/wxwin.m4:36: warning: underquoted definition of
AM_OPTIONS_WXCONFIG
  run info '(automake)Extending aclocal'
  or see
http://sources.redhat.com/automake/automake.html#Extending-aclocal
/usr/share/aclocal/wxwin.m4:59: warning: underquoted definition of AM_PATH_WXCONFIG
/usr/share/aclocal/linc.m4:1: warning: underquoted definition of AM_PATH_LINC
----

do they indicate that I did something wrong, or should I ignore them?

Continuing, I *did* ignore them and ran autoconf, then automake.
Automake complained about missing files, and to satisfy it I had to copy
the following files from the last release (2.5.33):

install-sh
missing
doc/mdate-sh
config.guess
config.sub
INSTALL
conf.in
ABOUT-NLS
depcomp

after which it ran without errors.  To get configure to run I also had
to copy

po/Makefile.in.in
po/remove-potcdate.sin

from the last release.  After doing all that, the build completed
without errors and the flex executable was created.  It was able to
report it's version, but I haven't tried it otherwise yet.

So my question is, what did I do wrong and what should I have done?

This is running on Gentoo Linux on x86.

Dustin

[Flex-devel] Re: alloca() problems in flex

[Herbert Y. <Her...@st...>]

Hi,

Currently I am trying to get flex to work under uClinux. Note that this is a 2.4.x Kernel with nommu and it uses the uClibc c library routines. When compiling I am having the problme of __builtin_alloca - namely alloca.c. 
It appears as though flex uses the function alloca() but wants to derive it from the kernel which is non-existant. I was wondering whether there was some sort of configuration in order to remove this or whether I could replace it/remove it completely and instead setup a static stack as uClinux only provides a static stack of 4K.

Thank you in advance for your help,
Herb


-- 
UTS CRICOS Provider Code:  00099F
DISCLAIMER: This email message and any accompanying attachments may contain
confidential information.  If you are not the intended recipient, do not
read, use, disseminate, distribute or copy this message or attachments.  If
you have received this message in error, please notify the sender immediately
and delete this message. Any views expressed in this message are those of the
individual sender, except where the sender expressly, and with authority,
states them to be the views the University of Technology Sydney. Before
opening any attachments, please check them for viruses and defects.