flex-help — help with using flex in other applications

[Flex-help] flex Manual Enhacement Suggestions

[Arthur S. <asc...@at...>]

I just started with flex and I wonder if you would consider some manual changes. As an addendum I am included my personal 'cheat sheet'. I doubt that I will ever change it and it (probably) severely handicapped with errors, but it gives an indication of my own focus. If it has any use, use it, otherwise, don't.


In the end each 'suggestion' 
should be prefixed or suffixed with the statement "would it be possible 
to". Since taste is subjective the list of things only reflect my own 
understanding and not that of the developers. Whatever comes of the 
suggestion the product is an outstandingly good one. Thanks.

art


There is no standardized capitalization rules for anything. Do you see a way forward where you could provide a standard for capitalization? As an example, in Section 10 you have the initial start condition noted as INITIAL and the start condition in the example noted as 'expect'. Why not EXPECT to make capitalization consistent with INITIAL?

Section 6 Patterns
There is no indication of what the valid range of characters is so there is no way to determined what {^x] means. I have assumed that the range is [:print:] so that [^x} translates to [:print:]{-}x or [\x21-\x7E]{-}x.

The character '*' seems to have two independent meanings, <sc>a* is different from [a*]. If this is correct it would be nice to include some statement to this effect in the manual.

You have specified that '.' represents any character except a newline. Does this mean that [^.] represents the range [\x00-\xFF]{-}\n, [:print:]{-}\n,[:graph:]{-}\n or ...? 


flex supports either a 7bit or an 8bit mode. What effect does either mode have on allowable patterns? Are the patterns restricted to [\x00-\x7E] so that they work in both modes? I think some statement concerning the universe of discourse and the effect on 7bit and 8bit is warranted.

There is no example on how to use [::] predefined ranges. Would it be possible to provide some? For example, in the definitions section:


is it
    NAME [[:print:]]     or

    NAME [{[:print:]}]   and/or

    NAME {[:print:]}     and/or

    NAME {[:print:]}     and/or

    NAME [:print:]

and similarly in the rules section.

The definition and example given for r/s should be made somewhat clearer. Perhaps as in:

 "If r and s are regular expressions, then r/s means that the
  lexer will recognize r only if s is present. If s is not 

  present then scanning will continue."

 "The regular expression 's' should be of fixed size. That is if 's '

  stands for a* where we are asking for an indefinite number of a's,
  then flex will/will not be able to generate a lexer."

 "In a similar way, if 's' is [abc|defg], where the alternatives are
  not the same size, flex will/will not be able to generate a lexer."
  Or the generated lexer will be ...

 "As an example, suppose we are generating a lexer for a mini-language,
  (right by the mini-bar in your hotel room) and we want all input
  numbers to be followed by a space. In other word, 123x will not be
  legal. We can represent this by:"

    [:digit]+/' '

 "Which translates to, follow one or more digits, [0-9], with a space."

 "Without the forward reference, that is [:digit:], the scanner will
  recognize a number followed by anything, for example, 'x', where the
  'x' will be scanned by the lexer as the beginning of some other
  token giving us two tokens, one for a number and one for 'x', instead
  of rejecting the input pattern."

  



Section 10 Start Conditions
The example on strings doesn't contain hexadecimal input. Is this deliberate?

In my lexer input files it seems that "<sc>{" will only work when '{' is adjacent to <sc>, that is, there is no space between the '>' and the '{'. If this is accurate do you think it should be mentioned?

Indices
In order of preference:
1: Would it be possible to include descriptive text on each value?
2: Could all descriptions be put into the same column, as in a table?

The alphabetized list serves a purpose for users who know the name of an item but do not remember it's full meaning or implementation. The current descriptive text allows a user to discover the organization of items but not the meaning of an item. Would it be possible to provide a separate set of indices organized functionally? This is useful to someone, like me, who is just starting out, doesn't know the names of items (so the alphabetized list has less meaning) but does understand the functional area where insight is needed, and with a little description, I'd be on my way.

I am including tables, below, that I constructed for my own use. I assure you that it is full of mistakes, is incomplete and should be subject to some loud laughs, but I think it covers most of flex in an acceptable manner. The first table is a template that I use to summarize various sections in the manual, and the second contains a summary of my understanding (and guesses) as to what a pattern is.

------------------------------------------------------
  Index of Multiple Input Buffer Functions and Macros
------------------------------------------------------

    ---------------------------------
              Functions
    ---------------------------------
    yy_create_buffer(File *, int)        Creates an input buffer
    yy_delete_buffer(YY_BUFFER_STATE)    Reclaim buffer memory
    yy_flush_buffer()                    Forces YY_INPUT() on next scan
    yy_new_buffer(FILE *, int)           Creates an input buffer (C++)
    yy_scan_buffer(char *, yy_size_t)    Create a yy_size_t YY_BUFFER_STATE using char *
    yy_scan_bytes(const char *, int)     Copy int bytes of char * and Create a YY_BUFFER_STATE
    yy_scan_string(const char *)         Copy all bytes of char * and Create a YY_BUFFER_STATE
    yy_switch_to_buffer(YY_BUFFER_STATE) Switch input buffer
    yypush_buffer_state(YY_BUFFER_STATE) Pushes buffer onto stack
    yypop_buffer_state()                 Pops and deletes buffer from stack

    ---------------------------------
              Typedefs
    ---------------------------------
    yy_size_t                            Integer typedef

    ---------------------------------
              Variables
    ---------------------------------
    YY_CURRENT_BUFFER                    YY_BUFFER_STATE handle of top of stack


-------------------------------------------
     Index of Patterns
-------------------------------------------
r, s stand for any regular expression, called 'pattern' below.

The collating order is for ANSI-C characters {UTF-8} and the normal range of characters considered is \x21-\x7FE.

7-bit and 8-bit options have no affect in pattern recognition.

Whitespace is defined as a blank (" ") plus any of \a, \b, \f, \n, \r, \t, \v in the pattern.

When a pattern is matched it is immediately processed and the next character is processed as the first character in the current start condition. This has the effect that if we are checking for decimal numbers followed by a space, 123abc is matched as 123 and then abc.

Pattern matching starts at column 1 of the rules section in the input file to flex. Column 1 in the input file is defined as either the first column following a new line or the first column following the close of a start condition list, ('<sc,...>').

The character '*' has a dual role. Within a bracketed regular expression, [ ],
and quoted, "*", it stands for itself. Otherwise it indicates 0 or more instances
of a regular expression.

The character '.' in no place stands for itself except when quoted, "." or in [?s:r]. In
all other cases it indicates "any character in [\x00-\x7E]" except \n.

\                              Escape the next character
\"                             Match '"'
\<                             Match '<' (only at line 1)
\>                             Match '>' (only at line 1)
\0                             Match the binary value ANSI-C NUL
\a                             Match the binary value 
\b                             Match the binary value ANSI-C BEL
\f                             Match the binary value ANSI_C FF form feed
\n                             Match the binary value ANSI-C newline
\r                             Match the binary value ANSI-C carriage return
\t                             Match the binary value ANSI-C HT horizonal tab
\v                             Match the binary value ANSI-C VT vertical tab
\123                           Match the binary value in octal
\x123                          Match the binary value in hexadecimal
x                              Match the character 'x'
.                              Match any character except '\n'
[abc]                          Match any of an 'a', 'b', or 'c' character
[a-m]                          Match any character from 'a' to 'm'
[abj-z]                        Match any of 'a', 'b', and 'j' through 'z'
[^abc]                         Match any character in [:ptrint:] except 'a', 'b', or 'c'
[^abc-m]                       Match any character in [:ptrint:] except 'a', 'b', or 'c' through 'm'
[a-z]{-}b                      Match any character except 'b'
r{-}s                          Exclude pattern s from pattern r
[a-z]{-}[aeiou]                Match any character except the vowels
[a-z]{-}[m-p]                  Match any character except any of 'm' through 'p'
r{+}s                          Include pattern s into pattern r
[a-z]{+}\n                     Match lower case characters and a new line
[a-z]{+}[0-9]                  Match lower case characters and numbers
"text"                         Match the literal string "text"
[: :]                          Predefined pattern
[:^ :]                         Negation of predefined pattern (everything except)
[:alnum:]                      Match [a-zA-Z0-9]
[:alpha:]                      Match [a-zA-Z]
[:blank:]                      Match [ \t]
[:cntrl:]                      Match [\x00-\x1F\x7F]
[:digit:]                      Match [0-9]
[:graph:]                      Match [\x21-\x7E]
[:lower:]                      Match [a-z]
[:print:]                      Match [\x20-\x7E]
[:punct:]                      Match [!"#$%&'()*+,\-./:;<=>?@\[\\\]^_`{|}~]
[:space:]                      Match [ \f\n\r\t\v]
[:upper:]                      Match [A-Z]
[:xdigit:]                     Match [a-fA-F0-9]


Each of the above is a regular expression.
{NAME}                         Reference to NAME surrounded by '(' ')'
r?                             Match the pattern 'r' 0 or 1 times
r+                             Match the pattern 'r' 1 or more times
r*                             Match the pattern 'r' 0 or more times
r{4}                           Match the pattern 'r' exactly 4 times
r{2,}                          Match the pattern 'r' 2 or more times
r{2,4}                         Match the pattern 'r' 2, 3, or 4 times
(r)                            Match the pattern 'r' and override precedence
rs                             Match pattern 'r' followed by 's' - concatenation
r|s                            Match either pattern 'r' or pattern 's'
r/s                            Match pattern 'r' but only if followed by 's'
^r                             Match pattern 'r' starting at the beginning of a line
r$                             Match pattern 'r' at the end of a line
<<EOF>>                        Match the end of the input file
(?o:r)                         Convert pattern 'r' according to the options 'o'
                               Note: the all negative options precede positive options
    i                          Case insensitive - [A] becomes [aA]
   -i                          Case sensitive   - [aA] becomes [aA]  default
    s                          Match any byte to '.'
   -s                          Match any byte except '\n' to '.'     default
    x                          Treat '/* */' as a comment and ignore whitespace
   -x                          Treat '/* */' as part of the pattern  default
                               do not ignore whitspace in patterns
   (?isx:r)                    Convert 'r', case insensitive, '.' matches anything
                               /* */ is part of the pattern
   (?x-is:r)                   Convert 'r', allow comments, do not change pattern
                               case, match '.' to any character except '\n'
(?# comment }                  Embedded comment in pattern    

Re: [Flex-help] Using Dfa.c of flex

[Beenish R. <bee...@ho...>]

I want to integrate my dfa minimization algorithm in that. Is that possible? Secondly, I also want to get that dfa structure which is holding the dfa for any regular expression.

> Date: Sat, 22 Jun 2013 08:55:27 -0400
> From: wes...@gm...
> To: bee...@ho...
> CC: fle...@li...
> Subject: Re: [Flex-help] Using Dfa.c of flex
> 
> What do you want to use dfa.c for?
> 
> Flex's generated scanners are intended to work apart from the flex
> distribution. In particular, it's not necessary to have any flex source
> files around to use a generated scanner.
> 
> --Will
> 
> On Saturday, 22 June 2013, 12:33 pm +0500, Beenish Raza <bee...@ho...> wrote:
> 
> > I want to make use of dfa.c file present in flex. How can I make use of that. Is there any example available? Secondly, how lex.yy.c make use of these files dfa.c etc. There is no such function calls related to dfa in that if I am not wrong.  		 	   		  
> > ------------------------------------------------------------------------------
> > This SF.net email is sponsored by Windows:
> > 
> > Build for Windows Store.
> > 
> > http://p.sf.net/sfu/windows-dev2dev
> > _______________________________________________
> > Flex-help mailing list
> > Fle...@li...
> > https://lists.sourceforge.net/lists/listinfo/flex-help
> 
> -- 
> Will Estes (wl...@us...)
> Flex Project Maintainer
> http://flex.sourceforge.net/
 		 	   		  

Re: [Flex-help] Using Dfa.c of flex

[Will E. <wes...@gm...>]

Please keep this thread on the list.

--Will

On Saturday, 22 June 2013,  6:11 pm +0500, Beenish Raza <bee...@ho...> wrote:

> I want to integrate my dfa minimization algorithm in that. Is that possible? Secondly, I also want to get that dfa structure which is holding the dfa for any regular expression.
> 
> > Date: Sat, 22 Jun 2013 08:55:27 -0400
> > From: wes...@gm...
> > To: bee...@ho...
> > CC: fle...@li...
> > Subject: Re: [Flex-help] Using Dfa.c of flex
> > 
> > What do you want to use dfa.c for?
> > 
> > Flex's generated scanners are intended to work apart from the flex
> > distribution. In particular, it's not necessary to have any flex source
> > files around to use a generated scanner.
> > 
> > --Will
> > 
> > On Saturday, 22 June 2013, 12:33 pm +0500, Beenish Raza <bee...@ho...> wrote:
> > 
> > > I want to make use of dfa.c file present in flex. How can I make use of that. Is there any example available? Secondly, how lex.yy.c make use of these files dfa.c etc. There is no such function calls related to dfa in that if I am not wrong.  		 	   		  
> > > ------------------------------------------------------------------------------
> > > This SF.net email is sponsored by Windows:
> > > 
> > > Build for Windows Store.
> > > 
> > > http://p.sf.net/sfu/windows-dev2dev
> > > _______________________________________________
> > > Flex-help mailing list
> > > Fle...@li...
> > > https://lists.sourceforge.net/lists/listinfo/flex-help
> > 
> > -- 
> > Will Estes (wl...@us...)
> > Flex Project Maintainer
> > http://flex.sourceforge.net/
>  		 	   		  

-- 
Will Estes (wl...@us...)
Flex Project Maintainer
http://flex.sourceforge.net/

Re: [Flex-help] Using Dfa.c of flex

[Will E. <wes...@gm...>]

What do you want to use dfa.c for?

Flex's generated scanners are intended to work apart from the flex
distribution. In particular, it's not necessary to have any flex source
files around to use a generated scanner.

--Will

On Saturday, 22 June 2013, 12:33 pm +0500, Beenish Raza <bee...@ho...> wrote:

> I want to make use of dfa.c file present in flex. How can I make use of that. Is there any example available? Secondly, how lex.yy.c make use of these files dfa.c etc. There is no such function calls related to dfa in that if I am not wrong.  		 	   		  
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by Windows:
> 
> Build for Windows Store.
> 
> http://p.sf.net/sfu/windows-dev2dev
> _______________________________________________
> Flex-help mailing list
> Fle...@li...
> https://lists.sourceforge.net/lists/listinfo/flex-help

-- 
Will Estes (wl...@us...)
Flex Project Maintainer
http://flex.sourceforge.net/

[Flex-help] Using Dfa.c of flex

[Beenish R. <bee...@ho...>]

I want to make use of dfa.c file present in flex. How can I make use of that. Is there any example available? Secondly, how lex.yy.c make use of these files dfa.c etc. There is no such function calls related to dfa in that if I am not wrong.  		 	   		  

Re: [Flex-help] Please help me about linguistic typology

[Will E. <wes...@gm...>]

On Sunday, 16 June 2013,  8:53 pm +0100, Hugh Sasse <hg...@dm...> wrote:

> 
> On Sun, 16 Jun 2013, Daniel Janzon wrote:
> 
> > _______________________________________
> > From: Hugh Sasse [hg...@dm...]
> > 
> > >> sed -i 's/\(.*[^ \t]\)[ \t]*$/\1/''  your-file.lex   # Remove whitespace at end of line
> > >
> > >It is the -e option to take an expression.  See tutorials here:
> > >http://www.dblab.ece.ntua.gr/~george/sed/
> > 
> > The -i option edits the file in-place, no stdout output. Hence the backup... The original
> > poster mentioned ubuntu and -i works for gnu sed. In FreeBSD -i takes an argument,
> > but works similarly.
> > 
> Sorry for that, then.  I'd missed that, and now I see where
> Perl, et al, get -i from.  Thank you.
> 

Oddly, the -i option is newish in GNU sed. It's an instance of sed
borrowing from perl.

>         Hugh
> 
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by Windows:
> 
> Build for Windows Store.
> 
> http://p.sf.net/sfu/windows-dev2dev
> _______________________________________________
> Flex-help mailing list
> Fle...@li...
> https://lists.sourceforge.net/lists/listinfo/flex-help

-- 
Will Estes (wl...@us...)
Flex Project Maintainer
http://flex.sourceforge.net/

Re: [Flex-help] Please help me about linguistic typology

[Hugh S. <hg...@dm...>]

On Sun, 16 Jun 2013, Daniel Janzon wrote:

> _______________________________________
> From: Hugh Sasse [hg...@dm...]
> 
> >> sed -i 's/\(.*[^ \t]\)[ \t]*$/\1/''  your-file.lex   # Remove whitespace at end of line
> >
> >It is the -e option to take an expression.  See tutorials here:
> >http://www.dblab.ece.ntua.gr/~george/sed/
> 
> The -i option edits the file in-place, no stdout output. Hence the backup... The original
> poster mentioned ubuntu and -i works for gnu sed. In FreeBSD -i takes an argument,
> but works similarly.
> 
Sorry for that, then.  I'd missed that, and now I see where
Perl, et al, get -i from.  Thank you.

        Hugh

Re: [Flex-help] Please help me about linguistic typology

[Daniel J. <dan...@ed...>]

_______________________________________
From: Hugh Sasse [hg...@dm...]

>> Save a copy of you lex document (no joke!) and experiment with the following commands, which I haven't verified to work:
>
>That should be "tex" rather than "lex" document, I think.  It is that
>document you wish to process, and preserve in case things go wrong.

True! I meant tex.

>>
>> sed -i 's/\(.*[^ \t]\)[ \t]*$/\1/''  your-file.lex   # Remove whitespace at end of line
>
>It is the -e option to take an expression.  See tutorials here:
>http://www.dblab.ece.ntua.gr/~george/sed/

The -i option edits the file in-place, no stdout output. Hence the backup... The original
poster mentioned ubuntu and -i works for gnu sed. In FreeBSD -i takes an argument,
but works similarly.

Re: [Flex-help] Please help me about linguistic typology with Lex

[Will E. <wes...@gm...>]

On Sunday, 16 June 2013,  8:36 am +0200, t ly <tl...@li...> wrote:

> Hello Mr Westes,
> 
> With the advice of a friend, I start to learn to use LaTex instead of MS Word for my redaction. He said that it will take me little time at the beginning, but it will help me to save a lot of time after. I began to find that he's right :)
> 
> Today, i try to convert some of my work-in-progress doucments from Word to Latex; but, i have alot of useless whitespace in the converted documents. On a francophone forum of Ubuntu users, one member suggest me to use Lex, a much more powerful text processing tool, to deal with it; but nobody know exactly how to do these following tasks:
> 
> 1) Remove all useless whitespace  (spaces and tabs) at the end of the line.
> 2) Replace all tabs with a fixed number of spaces, 4 for example.
> 3) Put a single white after a punctuation marks [,;:!?]. 

Your items 1-3 are all easily doable with sed.

> 4) Restore the forgotten upper-cases (transform the letters which found after punctuation marks [.!?] to capital letters.
> 

This is also doable with sed, although it's not as easy as the previous
items; a bit of experimentation should get it the way you want though.
If you don't have a large number of documents to convert, you may find
it just easy to edit this out by hand. Sometimes, there are words that
should be in lower case after punctuation.


> So, I wonder if you'd be kind enough to give me some lines of code which could do that. Thank you in advance for any help you can provide!
> 
> Cordially,
>  		 	   		   		 	   		  

-- 
Will Estes (wl...@us...)
Flex Project Maintainer
http://flex.sourceforge.net/

Re: [Flex-help] Please help me about linguistic typology

[Hugh S. <hg...@dm...>]

On Sun, 16 Jun 2013, Daniel Janzon wrote:

> Hi!
> 
> Flex is not the right tool for this. It's way to complicated (and powerful). You're francophone friend should have mentions the tool sed. Sed will be you're friend.

You may also find awk will do the job.  More recent options are Perl, Python
Aand Ruby.  Some of this is a matter of personal taste. sed is a good choice
for many reasons, though I have found it difficult to debug at times.
> 
> Save a copy of you lex document (no joke!) and experiment with the following commands, which I haven't verified to work:

That should be "tex" rather than "lex" document, I think.  It is that
document you wish to process, and preserve in case things go wrong.
> 
> sed -i 's/\(.*[^ \t]\)[ \t]*$/\1/''  your-file.lex   # Remove whitespace at end of line

It is the -e option to take an expression.  See tutorials here:
http://www.dblab.ece.ntua.gr/~george/sed/

This ("Unix text processing") might be helpful:
http://oreilly.com/openbook/utp/

> sed -i 's/\t/    /g' your-file.lex # Replace tabs with four spaces
> sed -i 's/\([,.;:!?]\)[ \t]*/\1 /g'  # Put a single white space after punvtuation marks
> sed -i 's/\([.?!][ \t]*[a-zA-Z]\)/\U\1/' your-file.lex  # Fix forgotten upper case
> 
> I didn't test run the above commands but they will hopefully get you on track.
> 
> This is not a sed mailing list so do not expect an extended discussion about sed here.
> 
> All the best,
> Daniel

        HTH
        Hugh
> 
> ________________________________________
> From: t ly [tl...@li...]
> Sent: Sunday, June 16, 2013 8:32 AM
> To: Fle...@li...
> Subject: [Flex-help] Please help me about linguistic typology
> 
> Hello everybody,
> 
> With the advice of a friend, I start to learn to use LaTex instead of MS Word for my redaction. He said that it will take me little time at the beginning, but it will help me to save a lot of time after. I began to find that he's right :)
> 
> Today, i try to convert some of my work-in-progress doucments from Word to Latex; but, i have alot of useless whitespace in the converted documents. On a francophone forum of Ubuntu users, one member suggest me to use Lex, a much more powerful text processing tool, to deal with it; but nobody know exactly how to do these following tasks:
> 
> 1) Remove all useless whitespace  (spaces and tabs) at the end of the line.
> 2) Replace all tabs with a fixed number of spaces, 4 for example.
> 3) Put a single white after a punctuation marks [,;:!?].
> 4) Restore the forgotten upper-cases (transform the letters which found after punctuation marks [.!?] to capital letters.
> 
> So, I wonder if you'd be kind enough to give me some lines of code which could do that. Thank you in advance for any help you can provide!
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by Windows:
> 
> Build for Windows Store.
> 
> http://p.sf.net/sfu/windows-dev2dev
> _______________________________________________
> Flex-help mailing list
> Fle...@li...
> https://lists.sourceforge.net/lists/listinfo/flex-help
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by Windows:
> 
> Build for Windows Store.
> 
> http://p.sf.net/sfu/windows-dev2dev
> _______________________________________________
> Flex-help mailing list
> Fle...@li...
> https://lists.sourceforge.net/lists/listinfo/flex-help
> 

Re: [Flex-help] Please help me about linguistic typology

[Daniel J. <dan...@ed...>]

Hi!

Flex is not the right tool for this. It's way to complicated (and powerful). You're francophone friend should have mentions the tool sed. Sed will be you're friend.

Save a copy of you lex document (no joke!) and experiment with the following commands, which I haven't verified to work:

sed -i 's/\(.*[^ \t]\)[ \t]*$/\1/''  your-file.lex   # Remove whitespace at end of line
sed -i 's/\t/    /g' your-file.lex # Replace tabs with four spaces
sed -i 's/\([,.;:!?]\)[ \t]*/\1 /g'  # Put a single white space after punvtuation marks
sed -i 's/\([.?!][ \t]*[a-zA-Z]\)/\U\1/' your-file.lex  # Fix forgotten upper case

I didn't test run the above commands but they will hopefully get you on track.

This is not a sed mailing list so do not expect an extended discussion about sed here.

All the best,
Daniel

________________________________________
From: t ly [tl...@li...]
Sent: Sunday, June 16, 2013 8:32 AM
To: Fle...@li...
Subject: [Flex-help] Please help me about linguistic typology

Hello everybody,

With the advice of a friend, I start to learn to use LaTex instead of MS Word for my redaction. He said that it will take me little time at the beginning, but it will help me to save a lot of time after. I began to find that he's right :)

Today, i try to convert some of my work-in-progress doucments from Word to Latex; but, i have alot of useless whitespace in the converted documents. On a francophone forum of Ubuntu users, one member suggest me to use Lex, a much more powerful text processing tool, to deal with it; but nobody know exactly how to do these following tasks:

1) Remove all useless whitespace  (spaces and tabs) at the end of the line.
2) Replace all tabs with a fixed number of spaces, 4 for example.
3) Put a single white after a punctuation marks [,;:!?].
4) Restore the forgotten upper-cases (transform the letters which found after punctuation marks [.!?] to capital letters.

So, I wonder if you'd be kind enough to give me some lines of code which could do that. Thank you in advance for any help you can provide!
------------------------------------------------------------------------------
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev
_______________________________________________
Flex-help mailing list
Fle...@li...
https://lists.sourceforge.net/lists/listinfo/flex-help

[Flex-help] Please help me about linguistic typology

[t ly <tl...@li...>]

Hello everybody,

With the advice of a friend, I start to learn to use LaTex instead of MS Word for my redaction. He said that it will take me little time at the beginning, but it will help me to save a lot of time after. I began to find that he's right :)

Today, i try to convert some of my work-in-progress doucments from Word to Latex; but, i have alot of useless whitespace in the converted documents. On a francophone forum of Ubuntu users, one member suggest me to use Lex, a much more powerful text processing tool, to deal with it; but nobody know exactly how to do these following tasks:

1) Remove all useless whitespace  (spaces and tabs) at the end of the line.
2) Replace all tabs with a fixed number of spaces, 4 for example.
3) Put a single white after a punctuation marks [,;:!?]. 
4) Restore the forgotten upper-cases (transform the letters which found after punctuation marks [.!?] to capital letters.

So, I wonder if you'd be kind enough to give me some lines of code which could do that. Thank you in advance for any help you can provide! 		 	   		  

[Flex-help] DFA states computation

[Beenish R. <bee...@ho...>]

I want to compute the total number of DFA states for a certain regular expression using FLEX. Which C files or functions will help me to achieve this task using FLEX? 		 	   		  

Re: [Flex-help] flex-2.5.37 says WARNING: netinet/in.h: present but cannot be compiled

[Will E. <wes...@gm...>]

If you comment out that include, can you compile flex?

I'm heading to: what can the autoconf stuff do to work around this? And
maybe I need to update my autofoo tool chain too. THat might
help...although how old are we talking on your system?

--Will


On Wednesday, 17 April 2013,  6:29 pm -0400, Dennis Clarke <dc...@bl...> wrote:

> 
> So I have this older server I decided to build flex on .. and I saw this : 
> .
> .
> .
> checking netinet/in.h usability... no
> checking netinet/in.h presence... yes
> configure: WARNING: netinet/in.h: present but cannot be compiled
> configure: WARNING: netinet/in.h:     check for missing prerequisite headers?
> configure: WARNING: netinet/in.h: see the Autoconf documentation
> configure: WARNING: netinet/in.h:     section "Present But Cannot Be Compiled"
> configure: WARNING: netinet/in.h: proceeding with the compiler's result
> configure: WARNING:     ## ---------------------------------------------- ##
> configure: WARNING:     ## Report this to fle...@li... ##
> configure: WARNING:     ## ---------------------------------------------- ##
> checking for netinet/in.h... no
> checking regex.h usability... yes
> .
> .
> .
> 
> 
> After running "make" anyways .. I eventually see : 
> 
> /opt/csw/gcc4/bin/gcc  -mno-app-regs -mcpu=v8 -mtune=supersparc -mno-vis -m32 -g -D_TS_ERRNO -Dsun=1 -Dunix=1 -D__sun=1 -D__unix=1 -D__SunOS_5_8 -D__SVR4=1 -DSOLARIS2=8 -D_POSIX_C_SOURCE=199506L -D_POSIX_PTHREAD_SEMANTICS -D_LARGEFILE64_SOURCE  -L/usr/local/lib -o flex ccl.o dfa.o ecs.o scanflags.o gen.o main.o misc.o nfa.o parse.o scan.o skel.o sym.o tblcmp.o yylex.o options.o scanopt.o buf.o tables.o tables_shared.o filter.o regex.o lib/libcompat.a /usr/local/lib/libintl.so /usr/local/lib/libiconv.so -lc -R/usr/local/lib -lm 
> Undefined                       first referenced
>  symbol                             in file
> htonl                               tables.o
> htons                               tables.o
> ld: fatal: Symbol referencing errors. No output written to flex
> collect2: ld returned 1 exit status
> gmake[2]: *** [flex] Error 1
> gmake[2]: Leaving directory `/usr/local/build/flex-2.5.37_SunOS5.8_sparc.001'
> gmake[1]: *** [all-recursive] Error 1
> gmake[1]: Leaving directory `/usr/local/build/flex-2.5.37_SunOS5.8_sparc.001'
> gmake: *** [all] Error 2
> $ 
> 
> 
> Not too sure what to do with that.
> 
> Dennis 
> 
> 
> ------------------------------------------------------------------------------
> Precog is a next-generation analytics platform capable of advanced
> analytics on semi-structured data. The platform includes APIs for building
> apps and a phenomenal toolset for data science. Developers can use
> our toolset for easy data analysis & visualization. Get a free account!
> http://www2.precog.com/precogplatform/slashdotnewsletter
> _______________________________________________
> Flex-help mailing list
> Fle...@li...
> https://lists.sourceforge.net/lists/listinfo/flex-help

-- 
Will Estes (wl...@us...)
Flex Project Maintainer
http://flex.sourceforge.net/

[Flex-help] flex-2.5.37 says WARNING: netinet/in.h: present but cannot be compiled

[Dennis C. <dc...@bl...>]

So I have this older server I decided to build flex on .. and I saw this : 
.
.
.
checking netinet/in.h usability... no
checking netinet/in.h presence... yes
configure: WARNING: netinet/in.h: present but cannot be compiled
configure: WARNING: netinet/in.h:     check for missing prerequisite headers?
configure: WARNING: netinet/in.h: see the Autoconf documentation
configure: WARNING: netinet/in.h:     section "Present But Cannot Be Compiled"
configure: WARNING: netinet/in.h: proceeding with the compiler's result
configure: WARNING:     ## ---------------------------------------------- ##
configure: WARNING:     ## Report this to fle...@li... ##
configure: WARNING:     ## ---------------------------------------------- ##
checking for netinet/in.h... no
checking regex.h usability... yes
.
.
.


After running "make" anyways .. I eventually see : 

/opt/csw/gcc4/bin/gcc  -mno-app-regs -mcpu=v8 -mtune=supersparc -mno-vis -m32 -g -D_TS_ERRNO -Dsun=1 -Dunix=1 -D__sun=1 -D__unix=1 -D__SunOS_5_8 -D__SVR4=1 -DSOLARIS2=8 -D_POSIX_C_SOURCE=199506L -D_POSIX_PTHREAD_SEMANTICS -D_LARGEFILE64_SOURCE  -L/usr/local/lib -o flex ccl.o dfa.o ecs.o scanflags.o gen.o main.o misc.o nfa.o parse.o scan.o skel.o sym.o tblcmp.o yylex.o options.o scanopt.o buf.o tables.o tables_shared.o filter.o regex.o lib/libcompat.a /usr/local/lib/libintl.so /usr/local/lib/libiconv.so -lc -R/usr/local/lib -lm 
Undefined                       first referenced
 symbol                             in file
htonl                               tables.o
htons                               tables.o
ld: fatal: Symbol referencing errors. No output written to flex
collect2: ld returned 1 exit status
gmake[2]: *** [flex] Error 1
gmake[2]: Leaving directory `/usr/local/build/flex-2.5.37_SunOS5.8_sparc.001'
gmake[1]: *** [all-recursive] Error 1
gmake[1]: Leaving directory `/usr/local/build/flex-2.5.37_SunOS5.8_sparc.001'
gmake: *** [all] Error 2
$ 


Not too sure what to do with that.

Dennis 

Re: [Flex-help] Multiple file processing

[Gautier DI F. <gau...@gm...>]

2013/4/11 Chris verBurg <che...@gm...>

>
> My guess (based on the limited code snippet) would be that you don't
> completely reset the lexer state in between files.  Make sure you're
> calling at least yyrestart, and possibly also BEGIN(INITIAL).
>
> -Chris
>

I found the solution, I have to add XMLlex_destroy(); at the end of the
function.

Re: [Flex-help] Multiple file processing

[Chris v. <che...@gm...>]

My guess (based on the limited code snippet) would be that you don't
completely reset the lexer state in between files.  Make sure you're
calling at least yyrestart, and possibly also BEGIN(INITIAL).

-Chris



On Wed, Apr 10, 2013 at 3:29 PM, Gautier DI FOLCO <gau...@gm...
> wrote:

> Hi all,
>
> I want to process many files, so I add this at the end of my xml.l:
> int ParseXml(char *s)
> {
>     int err = 0;
>     XMLin = fopen(s, "r");
>
>     if (!XMLin) {
>         puts("Error on XML file opening\n");
>     } else {
>         XML_switch_to_buffer( XML_create_buffer( XMLin, YY_BUF_SIZE ) );
>         err = XMLparse();
>         XML_delete_buffer( YY_CURRENT_BUFFER );
>         fclose(XMLin);
>     }
>
>     return err;
> }
>
> On the first execution it works (it returns 0), but on the second call it
> fails (it returns 1).
> According to
> http://flex.sourceforge.net/manual/Multiple-Input-Buffers.htmlit
> should works!
>
> What do I do wrong?
>
> For your help,
> Thanks by advance.
>
> ------------------------------------------------------------------------------
> Precog is a next-generation analytics platform capable of advanced
> analytics on semi-structured data. The platform includes APIs for building
> apps and a phenomenal toolset for data science. Developers can use
> our toolset for easy data analysis & visualization. Get a free account!
> http://www2.precog.com/precogplatform/slashdotnewsletter
> _______________________________________________
> Flex-help mailing list
> Fle...@li...
> https://lists.sourceforge.net/lists/listinfo/flex-help
>

[Flex-help] Multiple file processing

[Gautier DI F. <gau...@gm...>]

Hi all,

I want to process many files, so I add this at the end of my xml.l:
int ParseXml(char *s)
{
    int err = 0;
    XMLin = fopen(s, "r");

    if (!XMLin) {
        puts("Error on XML file opening\n");
    } else {
        XML_switch_to_buffer( XML_create_buffer( XMLin, YY_BUF_SIZE ) );
        err = XMLparse();
        XML_delete_buffer( YY_CURRENT_BUFFER );
        fclose(XMLin);
    }

    return err;
}

On the first execution it works (it returns 0), but on the second call it
fails (it returns 1).
According to http://flex.sourceforge.net/manual/Multiple-Input-Buffers.htmlit
should works!

What do I do wrong?

For your help,
Thanks by advance.

[Flex-help] Filter for lineno

[MockF M. <moc...@ya...>]

Hi,



I had difficulty in building flex in Windows because there is no regex.h in MSVC. When I check the source code I find that only filter_fix_linedirs is using regex.

It seems one purpose is to adjust all "#line num xxx.l" but not "#line num xxx.c" where xxx.c is the generated file name. When there is any compiling error only line number for xxx.l is reported. Is where lineno for xxx.c can be reported? If not why don't we ignore it or remove it?

Thanks

[Flex-help] Buffer state stack and yy_scan_string

[<csa...@gm...>]

Hi all,

I am trying to use yy_scan_string on a program which also uses buffer
state stacks ({yypush, yypop}_buffer_state). According to the
documentation, yy_scan_string switches current buffer so a string is
parsed, but my desired behaviour is to get back to parsing the file
(using yypop?) once the string has been parsed.

I have seen some references on the web to using

yypush_buffer_state(yy_parse_string(...))

but I have noticed the inner yy_parse_string subsitutes the current
buffer state, which then gets pushed, and the original state gets lost
(no way to get back to file parsing).

The solution I have found is a little bit weird:

yypush_buffer_state (YY_CURRENT_BUFFER);
yy_scan_string (m.definition.c_str());

Thus first the *current* state is pushed (effectively creating a copy),
but then the yy_scan_string changes current buffer to the string.

It works, but just wanted to ask if somebody knows a better way to do
it.

Thanks!

Carlos

[Flex-help] help with parsing bibtex

[Rudra B. <rud...@ao...>]

Dear friends,
I am a novice. I want to parse bibtex file using flex/bison. A sample
bibtex is:

@Book{a1,
author="amook",
Title="ASR",
Publisher="oxf",
Year="2010",
Add="UK",
Edition="1",
}
@Article{a2,
Author="Rudra Banerjee",
Title={FeNiMo},
Publisher={P{\"R}B},
Issue="12",
Page="36690",
Year="2011",
Add="UK",
Edition="1",
}

and for parsing this I have written the following code:

%{
#include <stdio.h>
#include <stdlib.h>
%}

%{
char yylval;
int YEAR,i;
//char array_author[1000];
%}
%x author
%x title
%x pub
%x year
%%
@          			printf("\nNEWENTRY\n");
[a-zA-Z][a-zA-Z0-9]*   		{printf("%s",yytext);
					BEGIN(INITIAL);}
author=                 	{BEGIN(author);}
<author>\"[a-zA-Z\/.]+\"  	{printf("%s",yytext);
                           		BEGIN(INITIAL);}
year=                   	{BEGIN(year);}
<year>\"[0-9]+\"        	{printf("%s",yytext);
                        		BEGIN(INITIAL);}
title=                 		{BEGIN(title);}
<title>\"[a-zA-Z\/.]+\"  	{printf("%s",yytext);
                           		BEGIN(INITIAL);}
publisher=                 	{BEGIN(pub);}
<pub>\"[a-zA-Z\/.]+\"  		{printf("%s",yytext);
                           		BEGIN(INITIAL);}
[a-zA-Z0-9\/.-]+=        printf("ENTRY TYPE ");
\"                      printf("QUOTE ");
\{                      printf("LCB ");
\}                      printf(" RCB");
;                       printf("SEMICOLON ");
\n                      printf("\n");
%%

int main(){
  yylex();
//char array_author[1000];
//printf("%d%s",&i,array_author[i]);
i++;
return 0;
}

The problem is that I want to separate key and val in different
variables and store it in some place (may be array).
Can I have some insight?

[Flex-help] re-entrant scanner, noyywrap, wee bugette.

[John P. H. <jph...@gm...>]

I'm using Fedora 17 with 2.5.35.

The macro defined in response to %option noyywrap needs to be updated
to take an argument even though it expands to a constant.

Has this been fixed in the current release?

Thanks.

[Flex-help] stream based parser

[Bin Zhang(张. <bz...@Hi...>]

hi all
Here, I write a push parser for stream data (partial token). I just show my simple idea in the sample, and the code need further review.
Stream based parser and lexier are very useful for network data analysis, so I hope the feature can be merged to later release.
I need your help.

Good Luck
bz...@hi...<mailto:bz...@hi...>

[Flex-help] Why is yytext[] content (built up by calls to yymore() in previous rules) lost at EOF?

[Thom B. <th...@xm...>]

I don't know if I've stumbled across a bug in flex, or I'm just using it
incorrectly.

See also http://stackoverflow.com/questions/14418560 for a prettier version
of this same question.


When a scanner generated by gnu-flex encounters end-of-file, it loses the
content of yytext[] that was left there by yymore() calls in previous
rules. This erroneous behavior ony happens if YY_INPUT() is redefined.

This may be a bug in flex, but it seems likely that there's something
missing -- something else that a flex scanner definition should provide
when it redefines YY_INPUT().

I have tested using flex 2.5.35 on both Ubuntu 12.04.1 and on Windows 7. On
both systems, the scanner loses the yytext[] content at EOF if the text to
be scanned is provided via an explicit definition ofYY_INPUT().

Below is a sample flex scanner (flex-test.l) that is intended to read and
print HTML comments, even if the last comment is unterminated. It works
correctly when its input is provided via yy_scan_string(), but fails when
its input is instead provided by an explicit definition of YY_INPUT(). In
the sample code, #if's are used to select between the yy_scan_string() and
the YY_INPUT() implementations. Specifically, the expected output:

    Begin comment: <!--
    More comment:  <!--incomplete
    EOF comment:   <!--incomplete

appears if the scanner is built using

    flex --nounistd flex-test.l && gcc -DREDEFINE_YY_INPUT=0 lex.yy.c

But if the scanner is built using

    flex --nounistd flex-test.l && gcc -DREDEFINE_YY_INPUT=1 lex.yy.c

(changing the =0 to =1), then this incorrect output appears:

    Begin comment: <!--
    More comment:  <!--incomplete
    EOF comment:

Notice the absence of any comment text in that last line of output.

Here is the sample code:

/* A scanner demonstrating bad interaction between yymore() and <<EOF>>
 * when YY_INPUT() is redefined: specifically, yytext[] content is lost. */

%{
#include <stdio.h>

int yywrap(void) { return 1; }

#if REDEFINE_YY_INPUT

  #define MIN(a,b) ((a)<(b) ? (a) : (b))

  const char *source_chars;
  size_t source_length;

  #define set_data(s) (source_chars=(s), source_length=strlen(source_chars))

  size_t get_data(char *buf, size_t request_size) {
    size_t copy_size = MIN(request_size, source_length);
    memcpy(buf, source_chars, copy_size);
    source_chars += copy_size;
    source_length -= copy_size;
    return copy_size;
  }

  #define YY_INPUT(buf,actual,ask) ((actual)=get_data(buf,ask))

#endif

%}

%x COMM

%%

"<!--"          printf("Begin comment: %s\n", yytext); yymore();
BEGIN(COMM);
<COMM>[^-]+     printf("More comment:  %s\n", yytext); yymore();
<COMM>.         printf("More comment:  %s\n", yytext); yymore();
<COMM>--+\ *[>] printf("End comment:   %s\n", yytext); BEGIN(INITIAL);
<COMM><<EOF>>   printf("EOF comment:   %s\n", yytext); BEGIN(INITIAL);
return 0;

.               printf("Other:         %s\n", yytext);

<<EOF>>         printf("EOF:           %s\n", yytext); return 0;
%%

int main(int argc, char **argv) {
  char *text = "<!--incomplete";

  #if REDEFINE_YY_INPUT
    set_data(text);
    yylex();
  #else
    YY_BUFFER_STATE state = yy_scan_string(text);
    yylex();
    yy_delete_buffer(state);
  #endif
}

Re: [Flex-help] flex C++ parser is brittle in systems with more then one version of flex installed

[Aaron S. <aa...@se...>]

I strongly recommend against trying to emit a fixed path to the header
file. Try to edit your XCode configuration. It is reasonable to expect that
the program and include search paths are in fact in sync, or overridden in
a consistent way.


On Mon, Nov 26, 2012 at 4:16 PM, John Horigan <jo...@gl...> wrote:

> That works if I am working with command-line tools. I did eventually get
> XCode to use the correct version of FlexLexer.h. But the current
> architecture requires users to make sure that their program search path and
> their include search path are in sync. This probably works out of the box
> for Linux systems but not for XCode. I haven't tried it, but I bet there is
> extra work to get Visual Studio to find FlexLexer.h as well.
>
> Outputting a class header file along with the class code file would help
> flex be more robust in development environments other than unix
> command-line tools.
>
> -- john
>
>
> On Mon, Nov 26, 2012 at 2:37 AM, Will Estes <wes...@gm...> wrote:
>
> > Does gcc -I /usr/local/include not do what you want?
> >
> > On Sunday, 25 November 2012,  7:51 pm -0800, John Horigan <
> > jo...@gl...> wrote:
> >
> > > I am having a hard time using flex 2.5.37 installed at /usr/local on my
> > OS
> > > X Mountain Lion system because flex 2.5.35 is already present under
> /usr.
> > > Getting XCode to use flex 2.5.37 is not too hard, but getting it to use
> > > /usr/local/include/FlexLexer.h is proving difficult.
> > >
> > > I would like a feature that would allow the lexer to include the
> correct
> > > version of FlexLexer.h. I propose that the option
> > >
> > > ‘--header-file=FILE, %option header-file="FILE"’
> > >
> > > be allowed for C++ lexers. If the header option is present then flex
> > copies
> > > FlexLexer.h to the specified file and includes that file in the lexer.
> > You
> > > could even copy the correct lexer class name into the output instead of
> > > relying on a #define. I would be willing to work on a patch for this
> > change
> > > if developers are interested.
> > >
> > > -- john
> > >
> >
> ------------------------------------------------------------------------------
> > > Monitor your physical, virtual and cloud infrastructure from a single
> > > web console. Get in-depth insight into apps, servers, databases,
> vmware,
> > > SAP, cloud infrastructure, etc. Download 30-day Free Trial.
> > > Pricing starts from $795 for 25 servers or applications!
> > > http://p.sf.net/sfu/zoho_dev2dev_nov
> > > _______________________________________________
> > > Flex-help mailing list
> > > Fle...@li...
> > > https://lists.sourceforge.net/lists/listinfo/flex-help
> >
> > --
> > Will Estes (wl...@us...)
> > Flex Project Maintainer
> > http://flex.sourceforge.net/
> >
>
> ------------------------------------------------------------------------------
> Monitor your physical, virtual and cloud infrastructure from a single
> web console. Get in-depth insight into apps, servers, databases, vmware,
> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
> Pricing starts from $795 for 25 servers or applications!
> http://p.sf.net/sfu/zoho_dev2dev_nov
> _______________________________________________
> Flex-help mailing list
> Fle...@li...
> https://lists.sourceforge.net/lists/listinfo/flex-help
>