flex-devel — development of flex itself

Re: [Flex-devel] Flex Skeleton for non-C languages

[Will E. <wes...@gm...>]

Yes, this is possible.

On Tuesday,  4 February 2025,  4:28 pm +0100, Simon Schmid <sch...@gm...> wrote:

> Hello,
>
> I wrote a skeleton for Ruby to generate LALR(1) parsers in Ruby with Bison,
> I'm using a handwritten lexer however, but would like to use Flex to
> generate one to complete the chain.
>
> Could you tell me if it's technically possible to write a backend for a
> non-C language? I noticed there's already a fake Golang backend (
> https://github.com/westes/flex/blob/master/src/go-flex.skl).
>
> Thank you and best,
> Simon


> _______________________________________________
> Flex-devel mailing list
> Fle...@li...
> https://lists.sourceforge.net/lists/listinfo/flex-devel


--
Will Estes
wes...@gm...

[Flex-devel] Flex Skeleton for non-C languages

[Simon S. <sch...@gm...>]

Hello,

I wrote a skeleton for Ruby to generate LALR(1) parsers in Ruby with Bison,
I'm using a handwritten lexer however, but would like to use Flex to
generate one to complete the chain.

Could you tell me if it's technically possible to write a backend for a
non-C language? I noticed there's already a fake Golang backend (
https://github.com/westes/flex/blob/master/src/go-flex.skl).

Thank you and best,
Simon

Re: [Flex-devel] Flex - ZIP Download

[Will E. <wes...@gm...>]

We don't release a .zip file; that's a misfeature of github that is not possible to turn off.

The latest release of flex is 2.6.4, hence the latest release is shown as 2.6.4.

On Friday,  2 February 2024,  6:46 pm +1100, Damian McGuckin <da...@es...> wrote:

>
> Hi,
>
> I douwnloaded the latest .zip file from Github and it build quite nicely.
> It said it was 2.6.4.
>
> I also downloaded from the releases page
>
> 	flex-2.6.4.tar.gz
>
> and build it.
>
> So far, so good.
>
> However, the result of the latter against a ???.lex file is very
> different from the output of the former.
>
> Why is this the case?
>
> Also, why does the latest version from Github still say it is 2.6.4?
>
> Just curious.
>
> Thanks - Damian
>
>
> _______________________________________________
> Flex-devel mailing list
> Fle...@li...
> https://lists.sourceforge.net/lists/listinfo/flex-devel

--
Will Estes
wes...@gm...

[Flex-devel] Flex - ZIP Download

[Damian M. <da...@es...>]

Hi,

I douwnloaded the latest .zip file from Github and it build quite nicely.
It said it was 2.6.4.

I also downloaded from the releases page

 	flex-2.6.4.tar.gz

and build it.

So far, so good.

However, the result of the latter against a ???.lex file is very
different from the output of the former.

Why is this the case?

Also, why does the latest version from Github still say it is 2.6.4?

Just curious.

Thanks - Damian

Re: [Flex-devel] Push-lexer

[Jerome H. <jer...@gm...>]

Hi all,

Please review https://github.com/westes/flex/pull/476 and give me your
opinion.
I can provide you with two files generated with the old and the new
version, so you can see the diffs, if it's easier than generating them
yourselves.

Cheers,
Jérôme.

On Sat, Feb 6, 2021 at 5:11 PM Mighty Jo <mig...@gm...> wrote:

> Understood. Sounds like you have a better handle on it than I do!
>
> -Joe
>
> On Sat, Feb 6, 2021, 11:04 Jerome Hamm <jer...@gm...> wrote:
>
>> Hi Joe,
>>
>> Thank you for your answer.
>> I might have missed something, but I didn't find any code that yields,
>> even with a reentrant lexer. bison returns YYPUSH_MORE. But for the lexer
>> interactive mode uses getc and processes the buffer as soon as it detects a
>> '\n', whereas non-interactive reads blocks. So if I'm not mistaken
>> interactive mode blocks on getc and does not yield.
>>
>> Cheers,
>> Jérôme.
>>
>> On Sat, Feb 6, 2021 at 3:29 PM Mighty Jo <mig...@gm...> wrote:
>>
>>> Hi, Jérôme!
>>>
>>> I'm not a maintainer, but they are pretty cool about considering pull
>>> requests. Add tests for any new functionality and be sure your fork passes
>>> 'make distcheck' when your tree has been cleaned with 'git clean -xdf'.
>>>
>>> If I may ask a silly question: Have you looked at the "interactive"
>>> scanner option? It's meant for reading from character devices. The
>>> differences are mostly invisible to the caller of yylex().
>>>
>>> I don't remember whether the interactive scanner yields or blocks when
>>> the input device is empty.
>>>
>>> Cheers!
>>> Joe
>>>
>>> On Sat, Feb 6, 2021, 08:12 Jerome Hamm <jer...@gm...> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am working on the ability to push data byte by byte (or more, but not
>>>> necessarily aligned to tokens or complete lines) to flex, so platforms
>>>> which don't have threads, like micro controllers can also benefit from it,
>>>> in a "cooperative-multitasking" spirit.
>>>>
>>>> What is the policy about adding new functionality to flex, would this
>>>> be something that could be integrated?
>>>>
>>>> Cheer,
>>>> Jérôme.
>>>> _______________________________________________
>>>> Flex-devel mailing list
>>>> Fle...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/flex-devel
>>>>
>>> _______________________________________________
>>> Flex-devel mailing list
>>> Fle...@li...
>>> https://lists.sourceforge.net/lists/listinfo/flex-devel
>>>
>>

Re: [Flex-devel] Push-lexer

[Mighty Jo <mig...@gm...>]

Understood. Sounds like you have a better handle on it than I do!

-Joe

On Sat, Feb 6, 2021, 11:04 Jerome Hamm <jer...@gm...> wrote:

> Hi Joe,
>
> Thank you for your answer.
> I might have missed something, but I didn't find any code that yields,
> even with a reentrant lexer. bison returns YYPUSH_MORE. But for the lexer
> interactive mode uses getc and processes the buffer as soon as it detects a
> '\n', whereas non-interactive reads blocks. So if I'm not mistaken
> interactive mode blocks on getc and does not yield.
>
> Cheers,
> Jérôme.
>
> On Sat, Feb 6, 2021 at 3:29 PM Mighty Jo <mig...@gm...> wrote:
>
>> Hi, Jérôme!
>>
>> I'm not a maintainer, but they are pretty cool about considering pull
>> requests. Add tests for any new functionality and be sure your fork passes
>> 'make distcheck' when your tree has been cleaned with 'git clean -xdf'.
>>
>> If I may ask a silly question: Have you looked at the "interactive"
>> scanner option? It's meant for reading from character devices. The
>> differences are mostly invisible to the caller of yylex().
>>
>> I don't remember whether the interactive scanner yields or blocks when
>> the input device is empty.
>>
>> Cheers!
>> Joe
>>
>> On Sat, Feb 6, 2021, 08:12 Jerome Hamm <jer...@gm...> wrote:
>>
>>> Hi,
>>>
>>> I am working on the ability to push data byte by byte (or more, but not
>>> necessarily aligned to tokens or complete lines) to flex, so platforms
>>> which don't have threads, like micro controllers can also benefit from it,
>>> in a "cooperative-multitasking" spirit.
>>>
>>> What is the policy about adding new functionality to flex, would this be
>>> something that could be integrated?
>>>
>>> Cheer,
>>> Jérôme.
>>> _______________________________________________
>>> Flex-devel mailing list
>>> Fle...@li...
>>> https://lists.sourceforge.net/lists/listinfo/flex-devel
>>>
>> _______________________________________________
>> Flex-devel mailing list
>> Fle...@li...
>> https://lists.sourceforge.net/lists/listinfo/flex-devel
>>
>

Re: [Flex-devel] Push-lexer

[Jerome H. <jer...@gm...>]

Hi Joe,

Thank you for your answer.
I might have missed something, but I didn't find any code that yields, even
with a reentrant lexer. bison returns YYPUSH_MORE. But for the lexer
interactive mode uses getc and processes the buffer as soon as it detects a
'\n', whereas non-interactive reads blocks. So if I'm not mistaken
interactive mode blocks on getc and does not yield.

Cheers,
Jérôme.

On Sat, Feb 6, 2021 at 3:29 PM Mighty Jo <mig...@gm...> wrote:

> Hi, Jérôme!
>
> I'm not a maintainer, but they are pretty cool about considering pull
> requests. Add tests for any new functionality and be sure your fork passes
> 'make distcheck' when your tree has been cleaned with 'git clean -xdf'.
>
> If I may ask a silly question: Have you looked at the "interactive"
> scanner option? It's meant for reading from character devices. The
> differences are mostly invisible to the caller of yylex().
>
> I don't remember whether the interactive scanner yields or blocks when the
> input device is empty.
>
> Cheers!
> Joe
>
> On Sat, Feb 6, 2021, 08:12 Jerome Hamm <jer...@gm...> wrote:
>
>> Hi,
>>
>> I am working on the ability to push data byte by byte (or more, but not
>> necessarily aligned to tokens or complete lines) to flex, so platforms
>> which don't have threads, like micro controllers can also benefit from it,
>> in a "cooperative-multitasking" spirit.
>>
>> What is the policy about adding new functionality to flex, would this be
>> something that could be integrated?
>>
>> Cheer,
>> Jérôme.
>> _______________________________________________
>> Flex-devel mailing list
>> Fle...@li...
>> https://lists.sourceforge.net/lists/listinfo/flex-devel
>>
> _______________________________________________
> Flex-devel mailing list
> Fle...@li...
> https://lists.sourceforge.net/lists/listinfo/flex-devel
>

Re: [Flex-devel] Push-lexer

[Mighty Jo <mig...@gm...>]

Hi, Jérôme!

I'm not a maintainer, but they are pretty cool about considering pull
requests. Add tests for any new functionality and be sure your fork passes
'make distcheck' when your tree has been cleaned with 'git clean -xdf'.

If I may ask a silly question: Have you looked at the "interactive" scanner
option? It's meant for reading from character devices. The differences are
mostly invisible to the caller of yylex().

I don't remember whether the interactive scanner yields or blocks when the
input device is empty.

Cheers!
Joe

On Sat, Feb 6, 2021, 08:12 Jerome Hamm <jer...@gm...> wrote:

> Hi,
>
> I am working on the ability to push data byte by byte (or more, but not
> necessarily aligned to tokens or complete lines) to flex, so platforms
> which don't have threads, like micro controllers can also benefit from it,
> in a "cooperative-multitasking" spirit.
>
> What is the policy about adding new functionality to flex, would this be
> something that could be integrated?
>
> Cheer,
> Jérôme.
> _______________________________________________
> Flex-devel mailing list
> Fle...@li...
> https://lists.sourceforge.net/lists/listinfo/flex-devel
>

[Flex-devel] Push-lexer

[Jerome H. <jer...@gm...>]

Hi,

I am working on the ability to push data byte by byte (or more, but not
necessarily aligned to tokens or complete lines) to flex, so platforms
which don't have threads, like micro controllers can also benefit from it,
in a "cooperative-multitasking" spirit.

What is the policy about adding new functionality to flex, would this be
something that could be integrated?

Cheer,
Jérôme.

Re: [Flex-devel] #include <unistd.h> in middle of scanner: bad idea

[Kaz K. <938...@ky...>]

On 2020-08-06 21:26, Kang-Che Sung wrote:
> On Thu, Aug 6, 2020 at 1:37 AM Kaz Kylheku <938...@ky...> 
> wrote:
>> 
>> Flex generates a file which contains
>> a #include <unistd.h> long after user material. This seems to be for
>> the sake of having a declaration of isatty.
>> 
>> The problem with this is that the user code can #define macros that
>> can break the header. C programs have to be careful with what they
>> do before system headers are included.
> 
> System headers can always be overridden by user code. A prominent 
> example is
> feature test macros, which are defined before the inclusion of system 
> headers
> and can change the ABI used for standard library functions.

I was writing more from a standard C point of view in the above 
paragraph.
ISO C has no "feature test macros".

They are POSIX feature (imitated in other platforms). There is a 
specific
vocabulary of feature test macros. You cannot just make up any symbol.

So for instance -D_POSIX_C_SOURCE on the compiler command line has
a well defined effect in POSIX environments, due to being documented in
a standard. Whereas -D_BLUB_SOURCE, which I just made up, is undefined
behavior.

My comment wasn't meant to imply that an application must never under 
any
circumstances define any sort of macro prior to including a system 
header.

Ah right; in ISO C, there is a famous counterexample:

   #define NDEBUG
   #include <assert.h>

Again, documented by language, so okay.

> 
>> Here is small repro test case:
>> 
>>    %{
>> 
>>    #define pause blah[]
>> 
>>    %}
>> 
>>    %%
>> 
>> Result:
>> 
>>    $ flex lex.l ; cc lex.yy.c
>>    lex.l:3:15: error: declaration of ‘blah’ as array of functions
>>    #define pause blah[]
> 
> Why would anyone want to do that? Redefining a standard library symbol 
> makes
> the code unportable (except for local scope redefines).

The point is that that is a local scope define. My sample doesn't
even include any headers at all. Of course things like <stdio.h> are 
implied
but they are expected to be before #define pause.

The programmer doesn't expect <unistd.h> in there at all.

What breaks it is that Flex generates the code such that there is a 
surprise
#include <unistd.h> long after the #define pause.

The inclusion of <unistd.h> will not break if it precedes the #define 
pause.

The issue isn't standard library symbols; I just picked that for an easy
repro. Any macro whatsoever could potentially cause a problem.

I also don't care about "portable" as much as "not breaking on any of 
the half
dozen platforms I'm building for".

>> How about just an "extern int isatty(int)" somewhere?
> 
> Forward declaration can somewhat defeat the purpose of the header. A 
> library
> implementation might not declare the function exactly as described in 
> the
> standard document. It might contain other attributes or ABI keywords.

Indeed; that's not a good solution.

Probably, the best thing would be not to have that isatty business in 
there
at all, unless it's requested by some explicit option.

isatty is still being wastefully called even if YY_INPUT has redirected 
the
lexer to scan from a character string instead of a stream.

The isatty calls even happen if the lexer is generated in -B (batch) 
mode.
(Flex 2.6.4 on Ubuntu 18).

Aha! I have a nice workaround:

   #define YY_NO_UNISTD_H
   #undef isatty
   #define isatty(x) 0

Presto; I'm no longer including <unistd.h> myself, there is no complaint
about an undeclared isatty, and the numerous ioctls in the strace 
output:

    ioctl(0, TCGETS, {B38400 opost isig icanon echo ...}) = 0

are gone!

(IMHO, the above effect should be obtained by just %option batch or -B.)

And that also gives us one example to the "why would anyone redefine
a library symbol" question.

If <unistd.h> were included at the top, then we could let that include
be, and just do:

   #undef isatty
   #define isatty(x) 0

This trick is going to work fine on all the platforms I care about,
so it is de facto portable enough.

There are theoretical ways that could still break, but if it were
perpetrated before #include <unistd.h>, the probability of failure
would go to 100%. That's really why you don't want surprise header
inclusions after user-defined material.

Re: [Flex-devel] #include <unistd.h> in middle of scanner: bad idea

[Kang-Che S. <exp...@gm...>]

On Thu, Aug 6, 2020 at 1:37 AM Kaz Kylheku <938...@ky...> wrote:
>
> Flex generates a file which contains
> a #include <unistd.h> long after user material. This seems to be for
> the sake of having a declaration of isatty.
>
> The problem with this is that the user code can #define macros that
> can break the header. C programs have to be careful with what they
> do before system headers are included.

System headers can always be overridden by user code. A prominent example is
feature test macros, which are defined before the inclusion of system headers
and can change the ABI used for standard library functions.

> Here is small repro test case:
>
>    %{
>
>    #define pause blah[]
>
>    %}
>
>    %%
>
> Result:
>
>    $ flex lex.l ; cc lex.yy.c
>    lex.l:3:15: error: declaration of ‘blah’ as array of functions
>    #define pause blah[]

Why would anyone want to do that? Redefining a standard library symbol makes
the code unportable (except for local scope redefines).

> How about just an "extern int isatty(int)" somewhere?

Forward declaration can somewhat defeat the purpose of the header. A library
implementation might not declare the function exactly as described in the
standard document. It might contain other attributes or ABI keywords.

[Flex-devel] #include <unistd.h> in middle of scanner: bad idea

[Kaz K. <938...@ky...>]

Hi,

I'd like to report a sort of bug.

Flex generates a file which contains
a #include <unistd.h> long after user material. This seems to be for
the sake of having a declaration of isatty.

The problem with this is that the user code can #define macros that
can break the header. C programs have to be careful with what they
do before system headers are included.

The programmer typically places #include's in the %{ %} section,
and then the assumption is that those are all the headers there are.

Here is small repro test case:

   %{

   #define pause blah[]

   %}

   %%

Result:

   $ flex lex.l ; cc lex.yy.c
   lex.l:3:15: error: declaration of ‘blah’ as array of functions
   #define pause blah[]

I understand that the directive can be suppressed with YY_NO_UNISTD_H,
but that's a workaround discovered after things break.

How about just an "extern int isatty(int)" somewhere?

Or just plant it in this section:

   /* First, we deal with  platform-specific or compiler-specific issues. 
*/

   /* begin standard C headers. */
   #include <stdio.h>
   #include <string.h>
   #include <errno.h>
   #include <stdlib.h>
   #ifndef YY_NO_UNISTD_H
   #include <unistd.h>
   #endif

   /* end standard C headers. */

If you're going to include <unistd.h> eventually, why beat around the
bush; clump it with other system headers.

If someone wants to disable that, they can use flex --nounistd.

Cheers ...

Re: [Flex-devel] filter.c: malloc+memset -> calloc

[Will E. <wes...@gm...>]

Thanks. This is on master and will be included in the next release of flex.

On Monday,  4 September 2017,  2:06 pm +0800, "Michael W. Bombardieri" <mb...@ii...> wrote:

> Hello,
> 
> In flex's filter.c two instances of memset() can be removed if calloc()
> is used for allocating the buffer. This was recently committed into
> OpenBSD's version of flex here:
> http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/lex/filter.c.diff?r1=1.7&r2=1.8&f=h
> 
> Have a nice day.
> 
> - Michael
> 
> 
> diff --git a/src/filter.c b/src/filter.c
> index ccdcdd6..a7e69ec 100644
> --- a/src/filter.c
> +++ b/src/filter.c
> @@ -47,10 +47,9 @@ struct filter *filter_create_ext (struct filter *chain, const char *cmd,
>  	va_list ap;
>  
>  	/* allocate and initialize new filter */
> -	f = malloc(sizeof(struct filter));
> +	f = calloc(sizeof(struct filter), 1);
>  	if (!f)
> -		flexerror(_("malloc failed (f) in filter_create_ext"));
> -	memset (f, 0, sizeof (*f));
> +		flexerror(_("calloc failed (f) in filter_create_ext"));
>  	f->filter_func = NULL;
>  	f->extra = NULL;
>  	f->next = NULL;
> @@ -100,10 +99,9 @@ struct filter *filter_create_int (struct filter *chain,
>  	struct filter *f;
>  
>  	/* allocate and initialize new filter */
> -	f = malloc(sizeof(struct filter));
> +	f = calloc(sizeof(struct filter), 1);
>  	if (!f)
> -		flexerror(_("malloc failed in filter_create_int"));
> -	memset (f, 0, sizeof (*f));
> +		flexerror(_("calloc failed in filter_create_int"));
>  	f->next = NULL;
>  	f->argc = 0;
>  	f->argv = NULL;
> 
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Flex-devel mailing list
> Fle...@li...
> https://lists.sourceforge.net/lists/listinfo/flex-devel

-- 
Will Estes
wes...@gm...

Re: [Flex-devel] filter_apply_chain() unused variable

[Will E. <wes...@gm...>]

This patch fails against master. Let me know if additional changes are needed from your perspective, otherwise I'll assume it's not needed at this point.

On Monday,  4 September 2017, 11:58 am +0800, "Michael W. Bombardieri" <mb...@ii...> wrote:

> Hello,
> 
> The variable r in flex's filter_apply_chain() is not needed.
> https://github.com/westes/flex/blob/master/src/filter.c#L175
> 
> It was recently removed from OpenBSD's in-tree version of flex.
> http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/lex/filter.c.diff?r1=1.8&r2=1.9&sortby=date&f=h
> 
> I'm posting it here in case it's considered useful.
> 
> - Michael
> 
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Flex-devel mailing list
> Fle...@li...
> https://lists.sourceforge.net/lists/listinfo/flex-devel

-- 
Will Estes
wes...@gm...

Re: [Flex-devel] scanopt rework?

[Will E. <wes...@gm...>]

On Monday, 13 November 2017,  5:21 pm +0800, Kang-Che Sung <exp...@gm...> wrote:

> Now, I'm not sure if I'm allowed to make pull requests that modifies the
> scanopt interface to match one of the GNU interfaces above.

A pull request is a pull request. The usual metric is that the more you change, the more important it has to be.

> My questions are these:
> 1. Will a pull request that may modify the optspec_t structure and the option.c
>    entries be accepted?

Depends. I have nothing against such a pull request as such.

> 2. If accepts such a pull request, should I keep the description field in
>    optspec_t and the scanopt_usage() function when possible, or is it fine to
>    remove them altogether? (I personally prefers to remove, as they
>    unnecessary complicates the translation.)


When you make the pull request, just explain why you do what you do and why that's better -- if it's not obvious from "why".

-- 
Will Estes
Flex Project Maintainer
wes...@gm...
https://github.com/westes/flex

[Flex-devel] scanopt rework?

[Kang-Che S. <exp...@gm...>]

The scanopt.c and optspec_t interfaces in Flex, as far as I have seen,
resembles other option parsing inferfaces such as argp_parse and getopt_long
from GNU libc. However, IMHO the scanopt interface in Flex is a poorer design
comparing to the two aforementioned GNU interfaces.

Now, I'm not sure if I'm allowed to make pull requests that modifies the
scanopt interface to match one of the GNU interfaces above.

All three interface are required to maintantain a table of long options, which
I think can reside in option.c as before. The differences among the interfaces
are the table fields used.

The current scanopt uses this:

  struct optspec_t {
      const char *opt_fmt;
      int         r_val;
      const char *desc;
  };
  // Example entries:
  // {"--tables-file[=FILE]", OPT_TABLES_FILE, "write tables to FILE"},
  // {"-o FILE",              OPT_OUTFILE,     "specify output filename"}

  // Leading dash or dashes is required, and determines whether option is short
  // or long.
  // The presense of '[' or '=' in opt_fmt string determines whether the option
  // takes an optional or required argument.
  // The desc field is currently unused in Flex.

The Argp Parser uses this as option entry:

  struct argp_option {
      const char *name;
      int key;         // If isascii(key), this also specifies short option.
      const char *arg; // Descriptive name of the option argument, if the
                       // option accepts one. Shown in --help output.
      int flags;       // OPTION_ARG_OPTIONAL or OPTION_ALIAS
      const char *doc; // Description. Shown in --help output.
      int group;       // Option group number (uses for sorting in --help).
  };
  // Example entries:
  // {"tables-file", OPT_TABLES_FILE, "FILE", OPTION_ARG_OPTIONAL,
  //    "write tables to FILE",    0},
  // {"outfile",     'o',             "FILE", 0,
  //    "specify output filename", 0}

  // Pro:
  // Internally accepts '--help' option and can format and print the help text
  // from the table.
  // Cons:
  // 1. Complex interface. Requires and calls a user subroutine for further
  //    option processing instead of "returns when an option is found".
  // 2. Interopability with gettext is unknown and undocumented (dammit GNU!)

getopt_long() uses this as option entry:

  struct option {
      const char *name;
      int has_arg;     // no_argument, required_argument or optional_argument
      int *flag;       // If non-NULL, sets (*flag = val)
      int val;         // Return value.
  };
  // This structure specify long options only. Short options are specified the
  // same way as getopt() and not in this struct.
  // Example entries:
  // {"tables-file", optional_argument, 0, OPT_TABLES_FILE},
  // {"outfile",     required_argument, 0, 'o'}

  // Pro:
  // Simpler structure. Doesn't care about output of '--help' or '--version'.
  // Con:
  // The "flag" field is not so useful (a poor design from GNU).

I'm not proposing to switch Flex's argument parser to use one of the GNU
interfaces. However, I would like to reserve the opportunity to be compatible
with one of them. Flex's options structure has an unused "desc" field, and
"opt_fmt" required internal pre-processing to be useful in scanopt(), which is
ugly IMO.

My questions are these:
1. Will a pull request that may modify the optspec_t structure and the option.c
   entries be accepted?
2. If accepts such a pull request, should I keep the description field in
   optspec_t and the scanopt_usage() function when possible, or is it fine to
   remove them altogether? (I personally prefers to remove, as they
   unnecessary complicates the translation.)

Re: [Flex-devel] filter_apply_chain() unused variable

[Will E. <wes...@gm...>]

Thanks for your report.

If you're able, submitting this as a pull request against github makes the mechanics of putting the patch in move faster and easier.

On Monday,  4 September 2017, 11:58 am +0800, "Michael W. Bombardieri" <mb...@ii...> wrote:

> Hello,
> 
> The variable r in flex's filter_apply_chain() is not needed.
> https://github.com/westes/flex/blob/master/src/filter.c#L175
> 
> It was recently removed from OpenBSD's in-tree version of flex.
> http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/lex/filter.c.diff?r1=1.8&r2=1.9&sortby=date&f=h
> 
> I'm posting it here in case it's considered useful.
> 
> - Michael
> 
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Flex-devel mailing list
> Fle...@li...
> https://lists.sourceforge.net/lists/listinfo/flex-devel

-- 
Will Estes
Flex Project Maintainer
wes...@gm...
https://github.com/westes/flex

[Flex-devel] filter.c: malloc+memset -> calloc

[Michael W. B. <mb...@ii...>]

Hello,

In flex's filter.c two instances of memset() can be removed if calloc()
is used for allocating the buffer. This was recently committed into
OpenBSD's version of flex here:
http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/lex/filter.c.diff?r1=1.7&r2=1.8&f=h

Have a nice day.

- Michael


diff --git a/src/filter.c b/src/filter.c
index ccdcdd6..a7e69ec 100644
--- a/src/filter.c
+++ b/src/filter.c
@@ -47,10 +47,9 @@ struct filter *filter_create_ext (struct filter *chain, const char *cmd,
 	va_list ap;
 
 	/* allocate and initialize new filter */
-	f = malloc(sizeof(struct filter));
+	f = calloc(sizeof(struct filter), 1);
 	if (!f)
-		flexerror(_("malloc failed (f) in filter_create_ext"));
-	memset (f, 0, sizeof (*f));
+		flexerror(_("calloc failed (f) in filter_create_ext"));
 	f->filter_func = NULL;
 	f->extra = NULL;
 	f->next = NULL;
@@ -100,10 +99,9 @@ struct filter *filter_create_int (struct filter *chain,
 	struct filter *f;
 
 	/* allocate and initialize new filter */
-	f = malloc(sizeof(struct filter));
+	f = calloc(sizeof(struct filter), 1);
 	if (!f)
-		flexerror(_("malloc failed in filter_create_int"));
-	memset (f, 0, sizeof (*f));
+		flexerror(_("calloc failed in filter_create_int"));
 	f->next = NULL;
 	f->argc = 0;
 	f->argv = NULL;

[Flex-devel] filter_apply_chain() unused variable

[Michael W. B. <mb...@ii...>]

Hello,

The variable r in flex's filter_apply_chain() is not needed.
https://github.com/westes/flex/blob/master/src/filter.c#L175

It was recently removed from OpenBSD's in-tree version of flex.
http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/lex/filter.c.diff?r1=1.8&r2=1.9&sortby=date&f=h

I'm posting it here in case it's considered useful.

- Michael

Re: [Flex-devel] api change for [yy]input()

[Nate R. <na...@go...>]

On Sun, Jun 25, 2017 at 12:34 PM Will Estes <wes...@gm...> wrote:

> On Monday, 22 May 2017, 11:10 pm +0000, Nate Rosenblum <na...@go...>
> wrote:
>
> > I recently discovered that the behavior of input() has changed from
> > returning EOF at end of file to returning 0. Searching around I came
> across
> > a closed github issue (https://github.com/westes/flex/issues/212), but
> this
> > really feels like something for the mailing list. So:
> >
> > Why change this return value, potentially breaking callers? Is it simply
> > because Apple chose to do so (
> >
> https://opensource.apple.com/source/flex/flex-26/patches/scanEOF.diff.auto.html
> )?
> > If that's the case, perhaps the man page should be updated to match.
>
> There was a reason why, but it escapes me. It's on my list and I keep
> thinking I will get to it and I keep not getting to it. I imagine this
> involves looking at the code and git blame in various ways.
>
> It was not solely because Apple. There was some particular thing that made
> me think it was a good idea.
>
> If you can point me to an example in the wild where this is a problem,
> that helps a bunch.
>
> Failing that, your point about updating the documentation is a good one --
> and pull requests against github are easy for me to process.
>

Thanks for getting back to me.

For my use case, the change is not a significant problem (just unexpected);
I already had to special-case the return value on OSX builds.

It seems like this change could, however, make it quite hard to use
yyinput() when scanning syntaxes that allow embedded null characters.
Returning -1 on end of file allows one to differentiate end of file
conditions from valid 0 bytes.

--nate

Re: [Flex-devel] api change for [yy]input()

[Will E. <wes...@gm...>]

On Monday, 22 May 2017, 11:10 pm +0000, Nate Rosenblum <na...@go...> wrote:

> I recently discovered that the behavior of input() has changed from
> returning EOF at end of file to returning 0. Searching around I came across
> a closed github issue (https://github.com/westes/flex/issues/212), but this
> really feels like something for the mailing list. So:
> 
> Why change this return value, potentially breaking callers? Is it simply
> because Apple chose to do so (
> https://opensource.apple.com/source/flex/flex-26/patches/scanEOF.diff.auto.html)?
> If that's the case, perhaps the man page should be updated to match.

There was a reason why, but it escapes me. It's on my list and I keep thinking I will get to it and I keep not getting to it. I imagine this involves looking at the code and git blame in various ways.

It was not solely because Apple. There was some particular thing that made me think it was a good idea.

If you can point me to an example in the wild where this is a problem, that helps a bunch.

Failing that, your point about updating the documentation is a good one -- and pull requests against github are easy for me to process.

[Flex-devel] api change for [yy]input()

[Nate R. <na...@go...>]

I recently discovered that the behavior of input() has changed from
returning EOF at end of file to returning 0. Searching around I came across
a closed github issue (https://github.com/westes/flex/issues/212), but this
really feels like something for the mailing list. So:

Why change this return value, potentially breaking callers? Is it simply
because Apple chose to do so (
https://opensource.apple.com/source/flex/flex-26/patches/scanEOF.diff.auto.html)?
If that's the case, perhaps the man page should be updated to match.

I did my best to search for discussion around December 2015 on this mailing
list but didn't see any; please feel free to just point me to an existing
thread if this has already been covered.

Best,

--nate

Re: [Flex-devel] GCC warning comparing signed and unsigned and possible bug

[Timothy A. <tim...@co...>]

On Thu, 2016-09-15 at 13:44 -0400, Will Estes wrote:
> You want to get your flex sources from github:
> 
> https://github.com/westes/flex/releases if you need a pre-built tar
> ball. Flex hasn't used cvs in years.

I see. It seems the latest master should no longer give this warning. 

Maybe you could update the sourceforge homepage with a link to the
github repo: http://flex.sourceforge.net/ I found no indications that
the project had move.

Thanks for all your work.

Tim

> 
> On Friday, 16 September 2016,  3:12 am +1000, Timothy Arceri <timothy
> .ar...@co...> wrote:
> 
> > 
> > On Thu, 2016-09-15 at 05:17 -0400, Will Estes wrote:
> > > 
> > > Which version of flex are you using? Looks like you've got an old
> > > version from the filenames you reference.
> > 
> > Version 2.6.0. The filenames are from a zipped copy of the source I
> > downloaded from sourceforges cvs system, they should be the latest
> > possible versions.
> > 
> > > 
> > > 
> > > On Thursday, 15 September 2016, 11:10 am +1000, Timothy Arceri
> > > <timot
> > > hy....@co...> wrote:
> > > 
> > > > 
> > > > 
> > > > Hi guys,
> > > > 
> > > > I'm seeing a GCC waring in the output from lex which seems to
> > > > be
> > > > caused
> > > > by the following line in flex.skl:
> > > > 
> > > >    YY_INPUT( (&YY_CURRENT_BUFFER_LVALUE-
> > > > > 
> > > > > yy_ch_buf[number_to_move]),
> > > > 			YY_G(yy_n_chars), num_to_read );
> > > > 
> > > > To fix it should be changed to:
> > > > 
> > > >    YY_INPUT( (&YY_CURRENT_BUFFER_LVALUE-
> > > > > 
> > > > > yy_ch_buf[number_to_move]),
> > > > 			YY_G(yy_n_chars), (unsigned)
> > > > num_to_read );
> > > > 
> > > > 
> > > > Something else I noticed is there are two flex.skl files, the
> > > > output
> > > > I'm getting seems to be generated by the one
> > > > in flex/to.do/unicode/flex.skl where num_to_read is defined as
> > > > int
> > > > which is fine. However the one in flex/flex.skl defines
> > > > num_to_read
> > > > as yy_size_t which is unsigned which means the while loop while
> > > > (
> > > > num_to_read <= 0 ) might not get executed, this looks like a
> > > > bug.
> > > > 
> > > > Tim
> > > > 
> > > > 
> > > > 
> > > > -------------------------------------------------------------
> > > > ----
> > > > -------------
> > > > _______________________________________________
> > > > Flex-devel mailing list
> > > > Fle...@li...
> > > > https://lists.sourceforge.net/lists/listinfo/flex-devel
> > > 
> 

Re: [Flex-devel] GCC warning comparing signed and unsigned and possible bug

[Will E. <wes...@gm...>]

You want to get your flex sources from github:

https://github.com/westes/flex/releases if you need a pre-built tar ball. Flex hasn't used cvs in years.

On Friday, 16 September 2016,  3:12 am +1000, Timothy Arceri <tim...@co...> wrote:

> On Thu, 2016-09-15 at 05:17 -0400, Will Estes wrote:
> > Which version of flex are you using? Looks like you've got an old
> > version from the filenames you reference.
> 
> Version 2.6.0. The filenames are from a zipped copy of the source I
> downloaded from sourceforges cvs system, they should be the latest
> possible versions.
> 
> > 
> > On Thursday, 15 September 2016, 11:10 am +1000, Timothy Arceri <timot
> > hy....@co...> wrote:
> > 
> > > 
> > > Hi guys,
> > > 
> > > I'm seeing a GCC waring in the output from lex which seems to be
> > > caused
> > > by the following line in flex.skl:
> > > 
> > >    YY_INPUT( (&YY_CURRENT_BUFFER_LVALUE-
> > > >yy_ch_buf[number_to_move]),
> > > 			YY_G(yy_n_chars), num_to_read );
> > > 
> > > To fix it should be changed to:
> > > 
> > >    YY_INPUT( (&YY_CURRENT_BUFFER_LVALUE-
> > > >yy_ch_buf[number_to_move]),
> > > 			YY_G(yy_n_chars), (unsigned) num_to_read );
> > > 
> > > 
> > > Something else I noticed is there are two flex.skl files, the
> > > output
> > > I'm getting seems to be generated by the one
> > > in flex/to.do/unicode/flex.skl where num_to_read is defined as int
> > > which is fine. However the one in flex/flex.skl defines num_to_read
> > > as yy_size_t which is unsigned which means the while loop while (
> > > num_to_read <= 0 ) might not get executed, this looks like a bug.
> > > 
> > > Tim
> > > 
> > > 
> > > 
> > > -----------------------------------------------------------------
> > > -------------
> > > _______________________________________________
> > > Flex-devel mailing list
> > > Fle...@li...
> > > https://lists.sourceforge.net/lists/listinfo/flex-devel
> > 

-- 
Will Estes
wes...@gm...

Re: [Flex-devel] GCC warning comparing signed and unsigned and possible bug

[Timothy A. <tim...@co...>]

On Thu, 2016-09-15 at 05:17 -0400, Will Estes wrote:
> Which version of flex are you using? Looks like you've got an old
> version from the filenames you reference.

Version 2.6.0. The filenames are from a zipped copy of the source I
downloaded from sourceforges cvs system, they should be the latest
possible versions.

> 
> On Thursday, 15 September 2016, 11:10 am +1000, Timothy Arceri <timot
> hy....@co...> wrote:
> 
> > 
> > Hi guys,
> > 
> > I'm seeing a GCC waring in the output from lex which seems to be
> > caused
> > by the following line in flex.skl:
> > 
> >    YY_INPUT( (&YY_CURRENT_BUFFER_LVALUE-
> > >yy_ch_buf[number_to_move]),
> > 			YY_G(yy_n_chars), num_to_read );
> > 
> > To fix it should be changed to:
> > 
> >    YY_INPUT( (&YY_CURRENT_BUFFER_LVALUE-
> > >yy_ch_buf[number_to_move]),
> > 			YY_G(yy_n_chars), (unsigned) num_to_read );
> > 
> > 
> > Something else I noticed is there are two flex.skl files, the
> > output
> > I'm getting seems to be generated by the one
> > in flex/to.do/unicode/flex.skl where num_to_read is defined as int
> > which is fine. However the one in flex/flex.skl defines num_to_read
> > as yy_size_t which is unsigned which means the while loop while (
> > num_to_read <= 0 ) might not get executed, this looks like a bug.
> > 
> > Tim
> > 
> > 
> > 
> > -----------------------------------------------------------------
> > -------------
> > _______________________________________________
> > Flex-devel mailing list
> > Fle...@li...
> > https://lists.sourceforge.net/lists/listinfo/flex-devel
>