jflex-users — from users for users, help, problems, discussions

Re: [jflex-users] What is wrong with the expression?

[Gerwin K. <ger...@ni...>]

Hi Kai,

the / operator can only be used in the rules part of the specification, not in 
macros.

This is to avoid it being nested (it is not really part of the regular languages).

Cheers,
Gerwin

Kai Hackemesser wrote:
> Hello,
> 
> In my .flex file I have defined following expression:
> 
> Name           = 
> [:letter:]+,{WhiteSpace}+([:letter:]+({WhiteSpace}+|\.))+/[^+]
> 
> this fails to parse at the '/' character close to the end. This is 
> intentionally there - a lookahead for a + char. Why fails the Generatoer 
> to parse this?
> 
> Cheers,
> Kai
> 
> 
> ------------------------------------------------------------------------------
> Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
> -OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
> -Strategies to boost innovation and cut costs with open source participation
> -Receive a $600 discount off the registration fee with the source code: SFAD
> http://p.sf.net/sfu/XcvMzF8H
> --
> jflex-users mailing list
> https://lists.sourceforge.net/lists/listinfo/jflex-users

[jflex-users] What is wrong with the expression?

[Kai H. <kai...@gm...>]

Hello,

In my .flex file I have defined following expression:

Name           = 
[:letter:]+,{WhiteSpace}+([:letter:]+({WhiteSpace}+|\.))+/[^+]

this fails to parse at the '/' character close to the end. This is 
intentionally there - a lookahead for a + char. Why fails the Generatoer 
to parse this?

Cheers,
Kai

[jflex-users] What is wrong with the expression?

[Kai H. <kai...@gm...>]

Hello,

In my .flex file I have defined following expression:

Name           = 
[:letter:]+,{WhiteSpace}+([:letter:]+({WhiteSpace}+|\.))+/[^+]

This fails to parse at the '/' character close to the end. This is 
intentionally there - a lookahead for a + char. Why fails the generator 
to parse this?

Cheers,
Kai

Re: [jflex-users] javascript grammar

[Marco T. <mar...@gm...>]

I also have some simple grammar written for a didactic project.

Marco

2009/2/18 Andreas Ravnestad <and...@gm...>

> > Would there general interest in setting up a grammar-contribution section
> with
> > different grammars that people have written?
> >
> > Cheers,
> > Gerwin
>
> I believe so. I have some simple grammars for simple search engine
> query parsing lying around.
>
> -Andreas
>
>
> ------------------------------------------------------------------------------
> Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco,
> CA
> -OSBC tackles the biggest issue in open source: Open Sourcing the
> Enterprise
> -Strategies to boost innovation and cut costs with open source
> participation
> -Receive a $600 discount off the registration fee with the source code:
> SFAD
> http://p.sf.net/sfu/XcvMzF8H
> --
> jflex-users mailing list
> https://lists.sourceforge.net/lists/listinfo/jflex-users
>

Re: [jflex-users] javascript grammar

[Andreas R. <and...@gm...>]

> Would there general interest in setting up a grammar-contribution section with
> different grammars that people have written?
>
> Cheers,
> Gerwin

I believe so. I have some simple grammars for simple search engine
query parsing lying around.

-Andreas

Re: [jflex-users] javascript grammar

[Gerwin K. <ger...@ni...>]

Hi Marco,

Marco Torello wrote:
> I would like to code a javascript grammar using jflex. Do you know if 
> there exist some of them?

I don't know of any, but it's quite possible that someone has done one already.

> I looked for them but I haven't found anything. Is it because nobody 
> needed it or because it is hard or even not possible to code?

I don't see why it shouldn't be possible. There is a Java grammar in the 
examples directory of JFlex, this may be a good starting point.

If you decide to write one and would like to re-distribute it, we could maybe 
arrange to include it on the JFlex web site.

Would there general interest in setting up a grammar-contribution section with 
different grammars that people have written?

Cheers,
Gerwin

[jflex-users] javascript grammar

[Marco T. <mar...@gm...>]

Hi,

I would like to code a javascript grammar using jflex. Do you know if there
exist some of them?
I looked for them but I haven't found anything. Is it because nobody needed
it or because it is hard or even not possible to code?

thank you for your help
Marco

[jflex-users] JFlex/CUP specification for bash needed urgently

[lizhun <liz...@ho...>]

Dear all, 
 
I need the JFlex/CUP specification sample file for simple bash language urgently, any help or clue will be very much appreciated, thanks in advance!With RegardsLi Zhun
_________________________________________________________________
MSN热搜榜，每天最In的信息资讯和热点排行让您一览无余！
 http://top.msn.com.cn

Re: [jflex-users] Look-behinds in JFlex?

[Gerwin K. <ger...@ni...>]

shef wrote:
> This might be an obvious one, but is there any way in JFlex to recognize an
> entity only if it is *preceded* by something else?

not built-in currently

> "a"/"b"
[..]
> What I want to do is recognize "b" only if it is preceded by "a", without
> consuming "a".

Interesting.

So what would the state be after this new expression has been matched? "b" 
should be removed from the input stream, but "a" should be left in? "a" should 
be available to be matched by the next round? Or is it enough to have both 
removed, but only "b" returned in yytext?

What's the application? (maybe we can solve it with something similar, but simpler)

Cheers,
Gerwin

[jflex-users] Look-behinds in JFlex?

[shef <sh...@ya...>]

This might be an obvious one, but is there any way in JFlex to recognize an entity only if it is *preceded* by something else?

You can do the opposite with lookaheads:

"a"/"b"

This recognizes "a" only if it is followed by "b". This doesn't consume "a".

What I want to do is recognize "b" only if it is preceded by "a", without consuming "a".

Is there any way to do this without writing a lot of Java code to keep track of the last token?



      

[jflex-users] flex generated code size is ten times as big as the jflex-generated code.

[Sunil B. <sun...@gm...>]

I ported a jflex file to flex.
but flex generated code size is ten times as big as the jflex-generated
code.

I wonder what makes this difference.

The following code is the jflex file.
------------------------------------------------------------------------------
import org.apache.lucene.analysis.Token;

%%

%class KoreanTokenizerImpl
%unicode
%integer
%function getNextToken
%pack
%char

%{

public static final int ALPHANUM          = 0;
public static final int APOSTROPHE        = 1;
public static final int ACRONYM           = 2;
public static final int COMPANY           = 3;
public static final int EMAIL             = 4;
public static final int HOST              = 5;
public static final int NUM               = 6;
public static final int CJ                = 7;
/**
 * @deprecated this solves a bug where HOSTs that end with '.' are
identified
 *             as ACRONYMs. It is deprecated and will be removed in the next
 *             release.
 */
public static final int ACRONYM_DEP       = 8;
public static final int KOREAN            = 9;

public static final String [] TOKEN_TYPES = new String [] {
    "<ALPHANUM>",
    "<APOSTROPHE>",
    "<ACRONYM>",
    "<COMPANY>",
    "<EMAIL>",
    "<HOST>",
    "<NUM>",
    "<CJ>",
    "<ACRONYM_DEP>",
    "<KOREAN>"
};

public final int yychar()
{
    return yychar;
}

/**
 * Fills Lucene token with the current token text.
 */
final void getText(Token t) {
  t.setTermBuffer(zzBuffer, zzStartRead, zzMarkedPos-zzStartRead);
}
%}

// korean word: a sequence of digits & letters &
KOREAN   = ({HANLETTER}|{LETTER}+ {HANLETTER}+  |{NUM}+ {HANLETTER}+
|{DIGIT}+ {HANLETTER}+ )+

// basic word: a sequence of digits & letters
ALPHANUM   = ({LETTER}|{DIGIT})+

// internal apostrophes: O'Reilly, you're, O'Reilly's
// use a post-filter to remove possesives
APOSTROPHE =  {ALPHA} ("'" {ALPHA})+

// acronyms: U.S.A., I.B.M., etc.
// use a post-filter to remove dots
ACRONYM    =  {LETTER} "." ({LETTER} ".")+

ACRONYM_DEP    = {ALPHANUM} "." ({ALPHANUM} ".")+

// company names like AT&T and Excite@Home.
COMPANY    =  {ALPHA} ("&"|"@") {ALPHA}

// email addresses
EMAIL      =  {ALPHANUM} (("."|"-"|"_") {ALPHANUM})* "@" {ALPHANUM}
(("."|"-") {ALPHANUM})+

// hostname
HOST       =  {ALPHANUM} ((".") {ALPHANUM})+

// floating point, serial, model numbers, ip addresses, etc.
// every other segment must have at least one digit
NUM        = ({ALPHANUM} {P} {HAS_DIGIT}
            | "." {DIGIT}
            | {DIGIT} "."
           | {HAS_DIGIT} {P} {ALPHANUM}
           | {ALPHANUM} ({P} {HAS_DIGIT} {P} {ALPHANUM})+
           | {HAS_DIGIT} ({P} {ALPHANUM} {P} {HAS_DIGIT})+
           | {ALPHANUM} {P} {HAS_DIGIT} ({P} {ALPHANUM} {P} {HAS_DIGIT})+
           | {HAS_DIGIT} {P} {ALPHANUM} ({P} {HAS_DIGIT} {P} {ALPHANUM})+)

// punctuation
P             = ("_"|"-"|"/"|"."|",")

// at least one digit
HAS_DIGIT  =
    ({LETTER}|{DIGIT})*
    {DIGIT}
    ({LETTER}|{DIGIT})*

ALPHA      = ({LETTER})+


LETTER     =
[\u0041-\u005a\u0061-\u007a\u00c0-\u00d6\u00d8-\u00f6\u00f8-\u00ff\u0100-\u1fff\uffa0-\uffdc]

DIGIT      =
[\u0030-\u0039\u0660-\u0669\u06f0-\u06f9\u0966-\u096f\u09e6-\u09ef\u0a66-\u0a6f\u0ae6-\u0aef\u0b66-\u0b6f\u0be7-\u0bef\u0c66-\u0c6f\u0ce6-\u0cef\u0d66-\u0d6f\u0e50-\u0e59\u0ed0-\u0ed9\u1040-\u1049]

HANLETTER     = [\uac00-\ud7af\u1100-\u11ff]

// Chinese, Japanese
CJ         =
[\u3040-\u318f\u3100-\u312f\u3040-\u309F\u30A0-\u30FF\u31F0-\u31FF\u3300-\u337f\u3400-\u4dbf\u4e00-\u9fff\uf900-\ufaff\uff65-\uff9f]

WHITESPACE = \r\n | [ \r\n\t\f]

%%

{KOREAN}                                                       { return
KOREAN; }
{ALPHANUM}                                                     { return
ALPHANUM; }
{APOSTROPHE}                                                   { return
APOSTROPHE; }
{ACRONYM}                                                      { return
ACRONYM; }
{COMPANY}                                                      { return
COMPANY; }
{EMAIL}                                                        { return
EMAIL; }
{HOST}                                                         { return
HOST; }
{NUM}                                                          { return NUM;
}
{CJ}                                                           { return CJ;
}
{ACRONYM_DEP}                                                  { return
ACRONYM_DEP; }



/** Ignore the rest */
. | {WHITESPACE}                                               { /* ignore
*/ }
------------------------------------------------------------------------------


and this is the flex file which I ported to
(unicode literal is replaced for test, because flex doesn't support unicode)

------------------------------------------------------------------------------
/* korean word: a sequence of digits & letters & */
KOREAN
({HANLETTER}|{LETTER}+{HANLETTER}+|{NUM}+{HANLETTER}+|{DIGIT}+{HANLETTER}+)+

/* basic word: a sequence of digits & letters */
ALPHANUM     ({LETTER}|{DIGIT})+

/* internal apostrophes: O'Reilly, you're, O'Reilly's
   use a post-filter to remove possesives
*/
APOSTROPHE    {ALPHA}(\x27\x00{ALPHA})+

/* acronyms: U.S.A., I.B.M., etc.
   use a post-filter to remove dots
*/
ACRONYM        {LETTER}\x2e\x00({LETTER}\x2e\x00)+

ACRONYM_DEP    {ALPHANUM}\x2e\x00({ALPHANUM}\x2e\x00)+

/* company names like AT&T and Excite@Home. */
COMPANY        {ALPHA}(\x26\x00|\x40\x00){ALPHA}

/* email addresses */
EMAIL
{ALPHANUM}((\x2e\x00|\x5f\x00|\x2d\x00){ALPHANUM})*\x40\x00{ALPHANUM}((\x2e\x00|\x2d\x00){ALPHANUM})+

/* hostname */
HOST        {ALPHANUM}((\x2e\x00){ALPHANUM})+

/* floating point, serial, model numbers, ip addresses, etc.
   every other segment must have at least one digit
*/
NUM
{ALPHANUM}{P}{HAS_DIGIT}|\x2e\x00{DIGIT}|{DIGIT}\x2e\x00|{HAS_DIGIT}{P}{ALPHANUM}|{ALPHANUM}({P}{HAS_DIGIT}{P}{ALPHANUM})+|{HAS_DIGIT}({P}{ALPHANUM}{P}{HAS_DIGIT})+|{ALPHANUM}{P}{HAS_DIGIT}({P}{ALPHANUM}{P}{HAS_DIGIT})+|{HAS_DIGIT}{P}{ALPHANUM}({P}{HAS_DIGIT}{P}{ALPHANUM})+

/* punctuation */
P            (\x5f\x00|\x2d\x00|\x2f\x00|\x2e\x00|\x2c\x00)

/* at least one digit */
HAS_DIGIT    ({LETTER}|{DIGIT})*{DIGIT}({LETTER}|{DIGIT})*

ALPHA        ({LETTER})+


/* for test */
LETTER        [abc]

DIGIT        [123]

HANLETTER    [def]

/* Chinese, Japanese */
CJ        [hij]


WHITESPACE    \x0d\x00\x0a\x00|\x0d\x00|\x0a\x00|\x09\x00|\x0c\x00|\x20\x00

%%

{KOREAN}                                                       {
printf("KOREAN\n"); }
{ALPHANUM}                                                     {
printf("ALPHANUM\n"); }
{APOSTROPHE}                                                   {
printf("APOSTROPHE\n"); }
{ACRONYM}                                                      {
printf("ACRONYM\n"); }
{COMPANY}                                                      {
printf("COMPANY\n"); }
{EMAIL}                                                        {
printf("EMAIL\n"); }
{HOST}                                                         {
printf("HOST\n"); }
{NUM}                                                          {
printf("NUM\n"); }
{CJ}                                                           {
printf("CJ\n"); }
{ACRONYM_DEP}                                                  {
printf("ACRONYM_DEP\n"); }
. | {WHITESPACE}                                               { /* ignore
*/ }

%%

Re: [jflex-users] dynamic scanning. Is it possible with jflex?

[Hadrien S. <had...@m4...>]

Hi Basil,

I've created a library in the past that provides this capability. It  
has APIs to create a JFlex and a Cup files, uses JFlex and Cup to  
generate a parser class with a different name each time, and loads it  
as the new parser. If you are interested, I'll look for it and send  
it in private. The problem is that it's old, not been recently  
compiled, and still had lots of bugs in it

Hadrien

Le 1 sept. 08 à 20:06, costa basil a écrit :

>
> Hi:
>
> Is it possible with JFlex to have a programming interface and do the
> scanning dynamically? So, instead of having JFlex generating the lexer
> (that is generating source code), JFlex would provide an interface
> that a programmer would use to set values, add actions and patterns,
> then JFlex does the magic and then I can run it all in one go without
> the need to generate any source code at all.
>
> Example of interface:
>
> Lets say you have an interface IAction that emulates an action that
> you'd normally specify in the lex file (such as: { return
> symbol(sym.ABSTRACT); })
>
> interface IAction {
>  public <actionReturnType> execute(<some parameters - could provide a
> reference to the lexer itelf>);
> }
>
>
> Then in the main lexer class (provided by JFlex) you have functions  
> such as:
>
> addPattern(String pattern, IAction action);
> addPattern(int state, String patter, IAction action);
>
> addDefinition (...) for lines such as: CommentContent = ( [^*] | \* 
> + [^/*] )*
>
> buildInternalTables()... - this does all the work to build tables or
> whatever that would be used in the scanning.
>
> The you are ready to do the parsing.
>
> One use for this would be to scan text files where the delimiters are
> configurable.
>
> If jflex can't do this, is there any other java tool that could have
> these features?
>
>
> Thanks
>
> ---------------------------------------------------------------------- 
> ---
> This SF.Net email is sponsored by the Moblin Your Move Developer's  
> challenge
> Build the coolest Linux based applications with Moblin SDK & win  
> great prizes
> Grand prize is a trip for two to an Open Source event anywhere in  
> the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> --
> jflex-users mailing list
> https://lists.sourceforge.net/lists/listinfo/jflex-users
>

Re: [jflex-users] dynamic scanning. Is it possible with jflex?

[costa b. <cos...@gm...>]

Hi:

Is it possible with JFlex to have a programming interface and do the
scanning dynamically? So, instead of having JFlex generating the lexer
(that is generating source code), JFlex would provide an interface
that a programmer would use to set values, add actions and patterns,
then JFlex does the magic and then I can run it all in one go without
the need to generate any source code at all.

Example of interface:

Lets say you have an interface IAction that emulates an action that
you'd normally specify in the lex file (such as: { return
symbol(sym.ABSTRACT); })

interface IAction {
 public <actionReturnType> execute(<some parameters - could provide a
reference to the lexer itelf>);
}


Then in the main lexer class (provided by JFlex) you have functions such as:

addPattern(String pattern, IAction action);
addPattern(int state, String patter, IAction action);

addDefinition (...) for lines such as: CommentContent = ( [^*] | \*+ [^/*] )*

buildInternalTables()... - this does all the work to build tables or
whatever that would be used in the scanning.

The you are ready to do the parsing.

One use for this would be to scan text files where the delimiters are
configurable.

If jflex can't do this, is there any other java tool that could have
these features?


Thanks

[jflex-users] symbols, integrating with CUP

[Dave P. <dav...@gm...>]

Trying to run the example at
http://www.jflex.de/manual.html#SECTION00040000000000000000

compiling gives many errors of the form

14. ERROR in Lexer.java (at line 766)
	return symbol(sym.STRING_LITERAL,
	              ^^^^^^^^^^^^^^^^^^
sym.STRING_LITERAL cannot be resolved
----------


I'm looking for a simpler example of integrating CUP with JFlex please.

I have both JFlex.jar and the CUP jar file on my classpath.

I *think* the sym.class contains the missing symbols but I'm not sure.

Any help appreciated.

TIA


-- 
Dave Pawson
XSLT XSL-FO FAQ.
http://www.dpawson.co.uk

Re: [jflex-users] merge more than one DFA into one

[Gerwin K. <ger...@ni...>]

I'm not sure what your question is. I doubt that anti-virus tools are
written with JFLex, but if they are, they just need to have each pattern
as a separate expression, not joined with |.

So, you write not "a|b  { some action }", but

a  { some action for a }
b  { some action for b }

If you are implementing your own DFA/regexp matcher, you need to
treat the final states with different actions separate in the minimisation
and NFA->DFA phase and keep track of what belongs to what.
Nothing magic there.

Cheers,
Gerwin

Johnson Wang wrote:
> But, if so, how does the anti-virus work? They must find the virus and detect what kind of virus they are. Is it possible to modify the structure of NFA/DFA to make it have that function? 
> 
> --------------------------------------------------
> From: "Gerwin Klein" <ger...@ni...>
> Sent: Monday, August 18, 2008 9:27 AM
> To: "Johnson Wang" <ins...@gm...>
> Cc: <jfl...@li...>
> Subject: Re: [jflex-users] merge more than one DFA into one
> 
>> Johnson Wang wrote:
>>> If I use JFlex to merge more than one DFA into one( use '|' connect each 
>>> other), then about the final DFA, how to know which regular 
>>> expression does the final states comes from? 
>>> For instance:
>>> Regular Expression1(r1):  a|b
>>> Regular Expression2(r2):  a|c
>>> We use '|' to connect these two regular expressions:  a|b|a|c
>>> Then, about the final states, how do we know the states comes from r1 or r2?
>> There is no general way to know. If you want to distinguish two expressions, you 
>> should match them separately in JFLex, otherwise the minimisation algorithm will 
>> collapse the final states into one.
>>
>> There is no speed penalty for matching them separately in JFlex (unless the two 
>> DFAs have thousands of state).
>>
>> Cheers,
>> Gerwin
>>
>>

Re: [jflex-users] merge more than one DFA into one

[Johnson W. <ins...@gm...>]

But, if so, how does the anti-virus work? They must find the virus and detect what kind of virus they are. Is it possible to modify the structure of NFA/DFA to make it have that function? 

--------------------------------------------------
From: "Gerwin Klein" <ger...@ni...>
Sent: Monday, August 18, 2008 9:27 AM
To: "Johnson Wang" <ins...@gm...>
Cc: <jfl...@li...>
Subject: Re: [jflex-users] merge more than one DFA into one

> Johnson Wang wrote:
>> If I use JFlex to merge more than one DFA into one( use '|' connect each 
>> other), then about the final DFA, how to know which regular 
>> expression does the final states comes from? 
>> For instance:
>> Regular Expression1(r1):  a|b
>> Regular Expression2(r2):  a|c
>> We use '|' to connect these two regular expressions:  a|b|a|c
>> Then, about the final states, how do we know the states comes from r1 or r2?
> 
> There is no general way to know. If you want to distinguish two expressions, you 
> should match them separately in JFLex, otherwise the minimisation algorithm will 
> collapse the final states into one.
> 
> There is no speed penalty for matching them separately in JFlex (unless the two 
> DFAs have thousands of state).
> 
> Cheers,
> Gerwin
> 
> 
> 

Re: [jflex-users] merge more than one DFA into one

[Gerwin K. <ger...@ni...>]

Johnson Wang wrote:
> If I use JFlex to merge more than one DFA into one( use '|' connect each 
> other), then about the final DFA, how to know which regular 
> expression does the final states comes from? 
> For instance:
> Regular Expression1(r1):  a|b
> Regular Expression2(r2):  a|c
> We use '|' to connect these two regular expressions:  a|b|a|c
> Then, about the final states, how do we know the states comes from r1 or r2?

There is no general way to know. If you want to distinguish two expressions, you 
should match them separately in JFLex, otherwise the minimisation algorithm will 
collapse the final states into one.

There is no speed penalty for matching them separately in JFlex (unless the two 
DFAs have thousands of state).

Cheers,
Gerwin

[jflex-users] merge more than one DFA into one

[Johnson W. <ins...@gm...>]

Dear Sirs,

If I use JFlex to merge more than one DFA into one( use '|' connect each other), then about the final DFA, how to know which regular expression does the final states comes from? 
For instance:
Regular Expression1(r1):  a|b
Regular Expression2(r2):  a|c
We use '|' to connect these two regular expressions:  a|b|a|c
Then, about the final states, how do we know the states comes from r1 or r2?

Tks for your time and help!

Re: [jflex-users] JFlex 1.4.2 issue

[Gerwin K. <ger...@ni...>]

Hi,

this looks like a bug. Could you please open a bug issue on the Sourceforge 
tracker and attach the spec (or a smaller one that triggers it)?

My first guess would be that I forgot to add macro expansion at some stage. 
Should be easy to fix.

Cheers,
Gerwin

shef wrote:
> Hello,
> 
> I just upgraded from 1.4.1 to 1.4.2 and ran JFlex against an existing file. I get this error:
> 
> java.lang.Error: When inserting lookahead expression: unkown expression type 42 in type = 42
> 
> I am using lookaheads in a couple of places. How do I debug this? How I isolate this error to a line in the code?
> 
> The lexer works fine when generated with 1.4.1, but I do get these warnings under 1.4.1:
> 
>     [jflex] Warning in file "<snip>...CommonLexer.flex" (line 386): 
>     [jflex] Dangerous lookahead expression, yytext might contain too many characters.
>     [jflex]     {xpath}/{space_equality}        { return TYPE_WORD; }  /* space_equality is a lookahead */
>     [jflex] Warning in file "<snip>...CommonLexer.flex" (line 387): 
>     [jflex] Dangerous lookahead expression, yytext might contain too many characters.
>     [jflex]     {xpath}/{contains}              { return TYPE_WORD; }
>     [jflex] Generated: CommonLexer.java
> 
> 
>       
> 
> -------------------------------------------------------------------------
> Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW!
> Studies have shown that voting for your favorite open source project,
> along with a healthy diet, reduces your potential for chronic lameness
> and boredom. Vote Now at http://www.sourceforge.net/community/cca08
> --
> jflex-users mailing list
> https://lists.sourceforge.net/lists/listinfo/jflex-users

[jflex-users] JFlex 1.4.2 issue

[shef <sh...@ya...>]

Hello,

I just upgraded from 1.4.1 to 1.4.2 and ran JFlex against an existing file. I get this error:

java.lang.Error: When inserting lookahead expression: unkown expression type 42 in type = 42

I am using lookaheads in a couple of places. How do I debug this? How I isolate this error to a line in the code?

The lexer works fine when generated with 1.4.1, but I do get these warnings under 1.4.1:

    [jflex] Warning in file "<snip>...CommonLexer.flex" (line 386): 
    [jflex] Dangerous lookahead expression, yytext might contain too many characters.
    [jflex]     {xpath}/{space_equality}        { return TYPE_WORD; }  /* space_equality is a lookahead */
    [jflex] Warning in file "<snip>...CommonLexer.flex" (line 387): 
    [jflex] Dangerous lookahead expression, yytext might contain too many characters.
    [jflex]     {xpath}/{contains}              { return TYPE_WORD; }
    [jflex] Generated: CommonLexer.java


      

[jflex-users] seeking sample codes

[shekar <drs...@gm...>]

seeking code for the paser and scanner Can any one help me out regarding
Formal Languages and compilers,

I want to write a code in Eclips(JFlex and CUP)(Scanner and parser)for
converting From *XML Schema to SQL DDL*.

so i want the sample code which takes the XML schema(XSD)and
scans,pars it produces the SQL DDL.

SO please tell me where can i find the sample code.

Re: [jflex-users] Strange jflex error

[Gerwin K. <ger...@ni...>]

Régis Cridlig wrote:
> Very true, Gerwin, my platform editor subreptitiously replaced the
> normal single quote  (0x27) by the Windows quote (character 0x92).

Ok, glad that is solved.


> BTW I am still confused by the JFlex User's Manual. At the bottom of
> page 8, you have:
>   \\\"                                                         {
> string.append('\"'); }
> why is the double quote preceded by the escape character in the action
> code? It does not make sense for a Java character.

True, that's not needed, just '"' is fine. I guess I wanted to avoid confusion 
with a string literal (which would be started with "), but apparently this leads 
to even more confusion ;-)

I should remove it from the manual.

Cheers,
Gerwin

Re: [jflex-users] Strange jflex error

[R. C. <cr...@gm...>]

Very true, Gerwin, my platform editor subreptitiously replaced the
normal single quote  (0x27) by the Windows quote (character 0x92).

BTW I am still confused by the JFlex User's Manual. At the bottom of
page 8, you have:
  \\\"                                                         {
string.append('\"'); }
why is the double quote preceded by the escape character in the action
code? It does not make sense for a Java character.

Thank you for your help,
-- Regis

2008/5/29 Gerwin Klein <ger...@ni...>:
> Hi Regis,
>
> I've investigated the problem, and it is not actually a problem with JFlex:
> the ' character after the string.append( is not the normal single quote
> (0x27), but 0x92 instead (my editor does not show a single quote there even,
> but I suspect that it is a character that looks very similar to ' on your
> platform). You might be able to reproduce this by copying&pasting the text
> from the bug report on
> https://sourceforge.net/tracker/?func=detail&atid=114929&aid=1978191&group_id=14929
> into a file and processing it with JFlex. It should work fine. If you
> download the attached file instead, you will get the error message
> described.
>
> It is possible that your platform maps the default single quote to 0x92, but
> you should have the same problem with normal Java programs then (JFlex uses
> the same mechanism to load and decode).
>
> I agree that the error message is obscure, but the one you would get from
> javac is not actually much better. I have tried replacing the correct ' with
> 0x92 in a .java file and I get from javac:
>
> Yylex.java:516: unclosed string literal
>          { string.append(?"?);
>                           ^
> Yylex.java:517: ';' expected
>          }
>          ^
> 2 errors
>
> Cheers,
> Gerwin
>

Re: [jflex-users] Strange jflex error

[Gerwin K. <ger...@ni...>]

Hi Regis,

I've investigated the problem, and it is not actually a problem with JFlex: the 
' character after the string.append( is not the normal single quote (0x27), but 
0x92 instead (my editor does not show a single quote there even, but I suspect 
that it is a character that looks very similar to ' on your platform). You might 
be able to reproduce this by copying&pasting the text from the bug report on
https://sourceforge.net/tracker/?func=detail&atid=114929&aid=1978191&group_id=14929
into a file and processing it with JFlex. It should work fine. If you download 
the attached file instead, you will get the error message described.

It is possible that your platform maps the default single quote to 0x92, but you 
should have the same problem with normal Java programs then (JFlex uses the same 
mechanism to load and decode).

I agree that the error message is obscure, but the one you would get from javac 
is not actually much better. I have tried replacing the correct ' with 0x92 in a 
.java file and I get from javac:

Yylex.java:516: unclosed string literal
           { string.append(?"?);
                            ^
Yylex.java:517: ';' expected
           }
           ^
2 errors

Cheers,
Gerwin

Re: [jflex-users] Strange jflex error

[Gerwin K. <ger...@ni...>]

Hi Regis,

thanks for the report. That should indeed work. I will investigate.

Cheers,
Gerwin

Régis Cridlig wrote:
> Hello!
> 
> I found this strange error in executing jflex 1.4.1. Same error with
> version 1.4.2:
> Reading "C:\Documents and Settings\Regis\My
> Documents\Info\workspace\BasicShell\strange.flex"
> 
> 
> Error in file "C:\Documents and Settings\Regis\My
> Documents\Info\workspace\BasicShell\strange.flex" (line 13):
> Unexpected character
> 
> \\\" 		 { string.append('"'); }
>                          ^
> 1 error, 0 warnings.
> 
> Generation aborted.
> 
> The source file is:
> /* strange.jflex
>  * Regis Cridlig
>  * 28 May 2008
>  */
> 
> %%
> 
> %{
>   StringBuffer string = new StringBuffer();
> %}
> 
> %%
> \\\" 		 { string.append('"'); }
> 
> I suspect a bug in the handling of the Java string.append('"'); block.
>  Could you help me?
> 
> Sincerely,
> 
> -- Regis Cridlig
> cr...@gm...
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> --
> jflex-users mailing list
> https://lists.sourceforge.net/lists/listinfo/jflex-users