jflex-users — from users for users, help, problems, discussions

Re: [jflex-users] Problem setting up keywords for SQL lexer

[Thomas K. <spa...@gm...>]

Peter,

thanks for the answer.

I already tried that (including \r\n to allow for multi-line) and basically that 
is working fine. The problem is, that the input string "PRIMARY KEYS" will 
return the wrong tokens: the first one being "PRIMARY KEY" and second one "S" - 
which is not correct.

I tried adding a whitespace after the last word "PRIMARY"{WS}"KEY"{WS} but then 
a token at the very end of the input string is not recognized, so I tried adding 
a word boundary "PRIMARY"{WS}"KEY"\b but that doesn't work either. It seems that 
the generated scanner handles the \b differently then Pattern class from Java.

Regards
Thomas


Peter L. Bird wrote on 09.07.2009 01:37:
> Thomas,
> 
> Here are pattern definitions that will get you a multi-word keyword to 
> be processed as a single token:
> 
> WS = [ \t]+
> 
> primaryKeyword = "PRIMARY"{WS}"KEY"
> 
> 
> This won't allow the keyword to be split across newline boundaries.  To 
> support that, you may need to use exclusive states.
> 
> peter
> 
> 
> At 05:32 PM 7/8/2009, Thomas Kellerer wrote:
>> Hi,
>>
>> I'm using a modified version of Stephen Ostermillers SQL Lexer that is 
>> part of
>> his syntax highlighting package (http://ostermiller.org/syntax)
>>
>> I want to add keywords that consists of more than one word e.g. 
>> "PRIMARY KEY"
>>
>> Currently the lexer definition contains something like this:
>>
>> keyword=("SELECT"|"UPDATE"|"DELETE")
>>
>> (apparently a very simplified version)
>>
>> Now I'm trying to add e.g. "PRIMARY KEY" as a single keyword to this 
>> list, but
>> the following will not work properly
>>
>> keyword=("SELECT"|"UPDATE"|"DELETE"|"PRIMARY KEY")
>>
>> as there may be any number of whitespace between PRIMARY and KEY and e.g.
>> PRIMARY    KEY would not be recognized by the Lexer. According to the
>> documentation (and my tests) the following will not work as well:
>>
>> keyword=("SELECT"|"UPDATE"|"DELETE"|"PRIMARY[ \t\r\n]+KEY")
>>
>> because the escape syntax is not interpreted in quoted strings.
>>
>> Leaving out the quotes is not working correctly either as e.g. PRIMARY 
>> KEYPOINT
>> will recognize PRIMARY KEY as a token as well.
>>
>> So my question is: how do I need to define multi-word keywords so that
>> whitespace is accepted but "partial" matches are processed correctly?
>>
>> I'm sorry, I'm using this somehow as a black-box as I'm not really 
>> experienced
>> with building lexers and all the theory behind it. So I'm maybe doing 
>> it the
>> wrong way or missing something very obvious.
>>
>> Any help is greatly appreciated
>>
>> Thanks
>> Thomas
>>
>>
>>
>> ------------------------------------------------------------------------------ 
>>
>> Enter the BlackBerry Developer Challenge
>> This is your chance to win up to $100,000 in prizes! For a limited time,
>> vendors submitting new applications to BlackBerry App World(TM) will have
>> the opportunity to enter the BlackBerry Developer Challenge. See full 
>> prize
>> details at: http://p.sf.net/sfu/Challenge
>> -- 
>> jflex-users mailing list
>> https://lists.sourceforge.net/lists/listinfo/jflex-users
> 

Re: [jflex-users] Problem setting up keywords for SQL lexer

[Peter L. B. <pb...@co...>]

Thomas,

Here are pattern definitions that will get you a multi-word keyword 
to be processed as a single token:

WS = [ \t]+

primaryKeyword = "PRIMARY"{WS}"KEY"


This won't allow the keyword to be split across newline 
boundaries.  To support that, you may need to use exclusive states.

peter


At 05:32 PM 7/8/2009, Thomas Kellerer wrote:
>Hi,
>
>I'm using a modified version of Stephen Ostermillers SQL Lexer that is part of
>his syntax highlighting package (http://ostermiller.org/syntax)
>
>I want to add keywords that consists of more than one word e.g. "PRIMARY KEY"
>
>Currently the lexer definition contains something like this:
>
>keyword=("SELECT"|"UPDATE"|"DELETE")
>
>(apparently a very simplified version)
>
>Now I'm trying to add e.g. "PRIMARY KEY" as a single keyword to this list, but
>the following will not work properly
>
>keyword=("SELECT"|"UPDATE"|"DELETE"|"PRIMARY KEY")
>
>as there may be any number of whitespace between PRIMARY and KEY and e.g.
>PRIMARY    KEY would not be recognized by the Lexer. According to the
>documentation (and my tests) the following will not work as well:
>
>keyword=("SELECT"|"UPDATE"|"DELETE"|"PRIMARY[ \t\r\n]+KEY")
>
>because the escape syntax is not interpreted in quoted strings.
>
>Leaving out the quotes is not working correctly either as e.g. 
>PRIMARY KEYPOINT
>will recognize PRIMARY KEY as a token as well.
>
>So my question is: how do I need to define multi-word keywords so that
>whitespace is accepted but "partial" matches are processed correctly?
>
>I'm sorry, I'm using this somehow as a black-box as I'm not really 
>experienced
>with building lexers and all the theory behind it. So I'm maybe doing it the
>wrong way or missing something very obvious.
>
>Any help is greatly appreciated
>
>Thanks
>Thomas
>
>
>
>------------------------------------------------------------------------------
>Enter the BlackBerry Developer Challenge
>This is your chance to win up to $100,000 in prizes! For a limited time,
>vendors submitting new applications to BlackBerry App World(TM) will have
>the opportunity to enter the BlackBerry Developer Challenge. See full prize
>details at: http://p.sf.net/sfu/Challenge
>--
>jflex-users mailing list
>https://lists.sourceforge.net/lists/listinfo/jflex-users

[jflex-users] Problem setting up keywords for SQL lexer

[Thomas K. <spa...@gm...>]

Hi,

I'm using a modified version of Stephen Ostermillers SQL Lexer that is part of
his syntax highlighting package (http://ostermiller.org/syntax)

I want to add keywords that consists of more than one word e.g. "PRIMARY KEY"

Currently the lexer definition contains something like this:

keyword=("SELECT"|"UPDATE"|"DELETE")

(apparently a very simplified version)

Now I'm trying to add e.g. "PRIMARY KEY" as a single keyword to this list, but
the following will not work properly

keyword=("SELECT"|"UPDATE"|"DELETE"|"PRIMARY KEY")

as there may be any number of whitespace between PRIMARY and KEY and e.g. 
PRIMARY    KEY would not be recognized by the Lexer. According to the 
documentation (and my tests) the following will not work as well:

keyword=("SELECT"|"UPDATE"|"DELETE"|"PRIMARY[ \t\r\n]+KEY")

because the escape syntax is not interpreted in quoted strings.

Leaving out the quotes is not working correctly either as e.g. PRIMARY KEYPOINT 
will recognize PRIMARY KEY as a token as well.

So my question is: how do I need to define multi-word keywords so that 
whitespace is accepted but "partial" matches are processed correctly?

I'm sorry, I'm using this somehow as a black-box as I'm not really experienced 
with building lexers and all the theory behind it. So I'm maybe doing it the 
wrong way or missing something very obvious.

Any help is greatly appreciated

Thanks
Thomas

[jflex-users] States on typing

[Stefan K. <ste...@gm...>]

Hi,

i use jflex for Syntax-Highlighting.
Is it alright, that it doesn't begin at YYINITIAL state after the first
sign?

Regards,
Stefan Kuhne

Re: [jflex-users] Switch into wrong state

[Stefan K. <ste...@gm...>]

Stefan Kuhne schrieb:
> Peter L. Bird schrieb:
>> It doesn't appear that you've issued the command "yybegin(nextState)" is
>> the initial block governed by the YYINITIAL state.  Therefore, you would
>> never get to the second block (in state SYMBOL_ERROR), but would end up
>> in the third block (assuming "=" is a symbol in {Error} )
>>
> So, with my little english.
> 
> I undersand you, like you say that i never come into the seccnd block
> (<SYMBOL, SYMBOL_ERROR> "=" ).
> But the first block (<YYINITIAL> {ErrorSym}) says, that i go to state
> "SYMBOL_ERROR" and not "YYINITIAL" or "COMMAND".
> 
> Where is my fault?
> 

I've found it.

<YYINITIAL> {ErrorSym} {
  nextState = SYMBOL_ERROR;
  lastToken = ASMToken.ERROR;
  String text = yytext();
  ASMToken t = (new ASMToken( lastToken, text, yyline, yychar, yychar
	+ text.length(), nextState));
  System.out.print("(SymbolError, " + text + ", " + nextState + ")");
  return (t);
}

I've fergot this:
yybegin(nextState);

Oh, you have marked it ;), but it was in my qoute :(.

Regards,
Stefan Kuhne

Re: [jflex-users] Switch into wrong state

[Stefan K. <ste...@gm...>]

Peter L. Bird schrieb:
> It doesn't appear that you've issued the command "yybegin(nextState)" is
> the initial block governed by the YYINITIAL state.  Therefore, you would
> never get to the second block (in state SYMBOL_ERROR), but would end up
> in the third block (assuming "=" is a symbol in {Error} )
> 
So, with my little english.

I undersand you, like you say that i never come into the seccnd block
(<SYMBOL, SYMBOL_ERROR> "=" ).
But the first block (<YYINITIAL> {ErrorSym}) says, that i go to state
"SYMBOL_ERROR" and not "YYINITIAL" or "COMMAND".

Where is my fault?

Regards,
Stefan Kuhne


> At 05:39 PM 6/27/2009, you wrote:
>> Hi,
>>
>> if a little mysterius Problem.
>> It come from:
>> <YYINITIAL> {ErrorSym} {
>>   nextState = SYMBOL_ERROR;
>>   lastToken = ASMToken.ERROR;
>>   String text = yytext();
>>   ASMToken t = (new ASMToken( lastToken, text, yyline, yychar, yychar
>> + text.length(), nextState));
>>   System.out.print("(SymbolError, " + text + ", " + nextState + ")");
>>   return (t);
>> }
>>
>> then it should go to:
>> <SYMBOL, SYMBOL_ERROR> "=" {
>>   nextState = OPERAND;
>>   lastToken = ASMToken.IDENTIFIER;
>>   String text = yytext();
>>   ASMToken t = (new ASMToken(lastToken, text, yyline, yychar, yychar
>> +  text.length(), nextState));
>>   System.out.print("(Symbol, " + text + ")");
>> *  yybegin(nextState);*
>>   return (t);
>> }
>>
>> but it go to:
>> <YYINITIAL, COMMAND> {Error} {
>>   System.out.print("(CommandError, " + nextState);
>>   nextState = COMMAND_ERROR;
>>   lastToken = ASMToken.ERROR;
>>   String text = yytext();
>>   ASMToken t = (new ASMToken(lastToken, text, yyline, yychar, yychar +
>> text.length(), nextState));
>>   System.out.print(", " + text + ")");
>> *  yybegin(nextState);*
>>   return(t);
>> }
>>
>> I can see it on:
>> (SymbolError, 09, 4): _ :(CommandError, 4, =): _ :
>>
>> The ": _ :" comes from whitespaces.
>>   public static final int SYMBOL_ERROR = 4;
>>

Re: [jflex-users] Switch into wrong state

[Peter L. B. <pb...@co...>]

It doesn't appear that you've issued the command "yybegin(nextState)" 
is the initial block governed by the YYINITIAL state.  Therefore, you 
would never get to the second block (in state SYMBOL_ERROR), but 
would end up in the third block (assuming "=" is a symbol in {Error} )

At 05:39 PM 6/27/2009, you wrote:
>Hi,
>
>if a little mysterius Problem.
>It come from:
><YYINITIAL> {ErrorSym} {
>   nextState = SYMBOL_ERROR;
>   lastToken = ASMToken.ERROR;
>   String text = yytext();
>   ASMToken t = (new ASMToken( lastToken, text, yyline, yychar, 
> yychar + text.length(), nextState));
>   System.out.print("(SymbolError, " + text + ", " + nextState + ")");
>   return (t);
>}
>
>then it should go to:
><SYMBOL, SYMBOL_ERROR> "=" {
>   nextState = OPERAND;
>   lastToken = ASMToken.IDENTIFIER;
>   String text = yytext();
>   ASMToken t = (new ASMToken(lastToken, text, yyline, yychar, 
> yychar +  text.length(), nextState));
>   System.out.print("(Symbol, " + text + ")");
>   yybegin(nextState);
>   return (t);
>}
>
>but it go to:
><YYINITIAL, COMMAND> {Error} {
>   System.out.print("(CommandError, " + nextState);
>   nextState = COMMAND_ERROR;
>   lastToken = ASMToken.ERROR;
>   String text = yytext();
>   ASMToken t = (new ASMToken(lastToken, text, yyline, yychar, 
> yychar + text.length(), nextState));
>   System.out.print(", " + text + ")");
>   yybegin(nextState);
>   return(t);
>}
>
>I can see it on:
>(SymbolError, 09, 4): _ :(CommandError, 4, =): _ :
>
>The ": _ :" comes from whitespaces.
>   public static final int SYMBOL_ERROR = 4;
>
>Regards,
>Stefan Kuhne
>
>
>
>
>
>------------------------------------------------------------------------------
>
>--
>jflex-users mailing list
>https://lists.sourceforge.net/lists/listinfo/jflex-users
>
>
>No virus found in this incoming message.
>Checked by AVG - www.avg.com
>Version: 8.5.375 / Virus Database: 270.12.93/2205 - Release Date: 
>06/27/09 05:53:00

[jflex-users] Switch into wrong state

[Stefan K. <ste...@gm...>]

Hi,

if a little mysterius Problem.
It come from:
<YYINITIAL> {ErrorSym} {
  nextState = SYMBOL_ERROR;
  lastToken = ASMToken.ERROR;
  String text = yytext();
  ASMToken t = (new ASMToken( lastToken, text, yyline, yychar,
yychar + 		text.length(), nextState));
  System.out.print("(SymbolError, " + text + ", " + nextState + ")");
  return (t);
}

then it should go to:
<SYMBOL, SYMBOL_ERROR> "=" {
  nextState = OPERAND;
  lastToken = ASMToken.IDENTIFIER;
  String text = yytext();
  ASMToken t = (new ASMToken(lastToken, text, yyline, yychar,
yychar + 			text.length(), nextState));
  System.out.print("(Symbol, " + text + ")");
  yybegin(nextState);
  return (t);
}

but it go to:
<YYINITIAL, COMMAND> {Error} {
  System.out.print("(CommandError, " + nextState);
  nextState = COMMAND_ERROR;
  lastToken = ASMToken.ERROR;
  String text = yytext();
  ASMToken t = (new ASMToken(lastToken, text, yyline, yychar,
yychar + 			text.length(), nextState));
  System.out.print(", " + text + ")");
  yybegin(nextState);
  return(t);
}

I can see it on:
(SymbolError, 09, 4): _ :(CommandError, 4, =): _ :

The ": _ :" comes from whitespaces.
  public static final int SYMBOL_ERROR = 4;

Regards,
Stefan Kuhne

Re: [jflex-users] Error rule in a JFlex state

[Peter L. B. <pb...@co...>]

The last rule of STATE1 will consume the first character of your 
input "NOBODY", and set the scanner into STATE2.  However, the 
pattern in STATE2 won't match because the input is now "OBODY" since 
the "N" was already consumed.  If you change the rule to be:

         .       { yybegin(STATE2);   yypushback(1); }

The function yypushback will push the character "N" from the matched 
text back onto the input stream to make it available for the pattern in STATE2.

At 09:26 AM 6/15/2009, Manos Krasoydakis wrote:

>Dear all,
>
>Setting the case that we have the following states:
>
><STATE1>
>{
>        "HELLO"                   {ACTION1}
>        "EVERYBODY"           {ACTION2}
>        .                              {yybegin(STATE2);}
>}
>
><STATE2>
>{
>         "NOBODY"                           {ACTION3}
>}
>
>I was wondering that if the scanner enters the STATE1 with current 
>input the word 'NOBODY' will it reach STATE2, and if yes will the 
>ACTION3 be taken? I would like to apologize if the question might 
>seem silly but I am newbie with JFlex.
>
>Thank you,
>Emmanouil.

[jflex-users] Error rule in a JFlex state

[Manos K. <agn...@ho...>]

Dear all,

Setting the case that we have the following states:

<STATE1>
{
       "HELLO"                   {ACTION1}
       "EVERYBODY"           {ACTION2}
       .                              {yybegin(STATE2);}
}

<STATE2>
{
        "NOBODY"                           {ACTION3}
}

I was wondering that if the scanner enters the STATE1 with current input the word 'NOBODY' will it reach STATE2, and if yes will the ACTION3 be taken? I would like to apologize if the question might seem silly but I am newbie with JFlex.

Thank you,
Emmanouil.

_________________________________________________________________
More than messages–check out the rest of the Windows Live™.
http://www.microsoft.com/windows/windowslive/

[jflex-users] SPARQL JFlex File

[<sch...@in...>]

hi everybody,

i'm currently working on a jflex file for SPARQL and i wonder if there
allready is one which i could use.

if you know of one please let me know.

-Segaja

Re: [jflex-users] 64K limit and source code

[Peter L. B. <pb...@co...>]

David,

This is, of course, a Java problem where the maximum size of all 
constants in a class is 64k.

I had a similar problem in the Java output for BYacc.  My solution 
was to wrap each of the larger tables in their own class, and make 
those tables STATIC within those classes.  The parse tables are then 
referenced using TableClass.table[ index ] notation.  If the tables 
are "readily identifiable", you can use a script to post-process the 
Parser.java file after running the parser generator.  You can even 
build a simple lexer with exclusive states to do this. :-) This is a 
simple addition to an Ant task (under an IDE like Eclipse).

peter bird

At 11:53 AM 5/28/2009, Dave Farr wrote:
>Hi All,
>
>I need to get passed the 64K limit that the action_table initializer in
>parser.java has broken. I believe the way to do this is to put the long
>string array into a text file. For that, I need to get the source code
>for CUP, set it up in JDeveloper and get it to run without errors. Can
>anyone help with this? (Or if anyone has a better idea on how to get
>around this problem, let me know).
>
>Thanks,
>Dave Farr
>
>------------------------------------------------------------------------------
>Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
>is a gathering of tech-side developers & brand creativity professionals. Meet
>the minds behind Google Creative Lab, Visual Complexity, Processing, &
>iPhoneDevCamp as they present alongside digital heavyweights like Barbarian
>Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com
>--
>jflex-users mailing list
>https://lists.sourceforge.net/lists/listinfo/jflex-users
>
>No virus found in this incoming message.
>Checked by AVG - www.avg.com
>Version: 8.5.339 / Virus Database: 270.12.43/2139 - Release Date: 
>05/28/09 08:10:00

[jflex-users] 64K limit and source code

[Dave F. <dav...@or...>]

Hi All,

I need to get passed the 64K limit that the action_table initializer in 
parser.java has broken. I believe the way to do this is to put the long 
string array into a text file. For that, I need to get the source code 
for CUP, set it up in JDeveloper and get it to run without errors. Can 
anyone help with this? (Or if anyone has a better idea on how to get 
around this problem, let me know).

Thanks,
Dave Farr

Re: [jflex-users] Newbie, need help with expression

[Leonardo G. <leo...@ma...>]

Hi,

 

            From the Jflex Manual:

 

~a (upto)

matches everything up to (and including) the first occurrence of a text
matched by a.

The expression ~a is equivalent to !([^]* a [^]*) a. A traditional C-style
comment

is matched by "/*" ~"*/"

 

So, if you´re looking for something that matches everything until “>” it
could be:

 

            ~”>”

 

I hope it works for you!

 

            Regards…

 

  _____  

From: Manos Krasoydakis [mailto:agn...@ho...] 
Sent: Domingo, 24 de Mayo de 2009 05:00 p.m.
To: jfl...@li...
Subject: [jflex-users] Newbie, need help with expression

 


Hi there,

I am quiet newbie with JFlex, and I would like to ask for some help. I want
to write an expression for matching a text-pattern until a symbol has been
found.
For example, setting the case that the input to the scanner is <HTML>, and
the symbol which when met the scanner has to stop is >, then the result
matched text will be <HTML. Can anybody please help me?

Thank you.

  _____  

See all the ways you can stay connected to
<http://www.microsoft.com/windows/windowslive/default.aspx>  friends and
family

[jflex-users] Newbie, need help with expression

[Manos K. <agn...@ho...>]

Hi there,

I am quiet newbie with JFlex, and I would like to ask for some help. I want to write an expression for matching a text-pattern until a symbol has been found.
For example, setting the case that the input to the scanner is <HTML>, and the symbol which when met the scanner has to stop is >, then the result matched text will be <HTML. Can anybody please help me?

Thank you.

_________________________________________________________________
Show them the way! Add maps and directions to your party invites. 
http://www.microsoft.com/windows/windowslive/products/events.aspx

[jflex-users] new to jflex, need help please

[Florin <gl...@ya...>]

I'm trying do a project that involves building a HTML grammar. Does anyone have the .flex file and  .cup (java cup)  file for this grammar already built by any chance? Please help :(



      

[jflex-users] I want to post to jflex list please

[Florin M. <gl...@ya...>]

My email is      gl...@ya...



      

Re: [jflex-users] ArrayIndexOutOfBounds exception in next_token()

[Gerwin K. <ger...@ni...>]

Hi Dave,

the error should go away if you use the %unicode directive in your spec.

Cheers,
Gerwin

Dave Farr wrote:
> Hi,
> 
>  
> 
> I'm try to parse a database specification from oracle. One of the lines 
> reads as follows:
> 
>  
> 
>  DESCRIPTION {This metric tracks the hourly rate for an employee. This 
> metric provides a snapshot of an employee’s pay rate that will be used 
> in employee information listings. Also, you can analyze hourly rates by 
> jobs, job families, positions, and pay grades to
> analyze your pay rate structure.
> }
> 
>  
> 
> The parser is set up to ignore everything between the braces.
> 
>  
> 
> We get an ArrayIndexOutOfBounds exception in next_token() when trying to 
> read the character '€' where zzInput == 226 and zzCMapL has a size of 128.
> 
>  
> 
> I would appreciate any insight into this.
> 
>  
> 
> Thanks,
> 
>  
> 
> David Farr
> 
>  
> 
> 
> ------------------------------------------------------------------------
> 
> ------------------------------------------------------------------------------
> Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are
> powering Web 2.0 with engaging, cross-platform capabilities. Quickly and
> easily build your RIAs with Flex Builder, the Eclipse(TM)based development
> software that enables intelligent coding and step-through debugging.
> Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com
> 
> 
> ------------------------------------------------------------------------
> 
> --
> jflex-users mailing list
> https://lists.sourceforge.net/lists/listinfo/jflex-users

Re: [jflex-users] How to remove println()s in JFlex generated class

[Gerwin K. <ger...@ni...>]

Hi James,

it sounds like your scanner spec is using either %standalone or %debug. These 
will generate the println that are you are mentioning. Unless you really want 
%standalone, you should leave these out.

Cheers,
Gerwin

James Cant wrote:
> Hi,
> 
> I hope someone maybe able to help me with a JFlex problem.
> 
> I have a string representation of a data structure and hope to tokenize 
> it with JFlex and then build the data structure from the token stream.  
> I started with the simple example in the distributed examples 
> directory.  I got it working and made some of the modifications I'll 
> need to do the job.
> 
> However, the generated scanner prints out a couple of lines each time 
> yylex() is called.  Though this is fine for the example, it is a problem 
> for my application.  How can I make these go away?  I didn't find a 
> solution in the manual nor did I see where these System.out.println's 
> originate in either the skeleton file or my .flex input.
> 
> I'd really appreciate your advice (or solution!).  Currently I'm 
> removing these lines with a perl script before compiling the JFlex 
> generated class; I view this as somewhat inelegant.
> Thanks for your interest.
> jim cant
> 
> ------------------------------------------------------------------------------
> Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are
> powering Web 2.0 with engaging, cross-platform capabilities. Quickly and
> easily build your RIAs with Flex Builder, the Eclipse(TM)based development
> software that enables intelligent coding and step-through debugging.
> Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com
> --
> jflex-users mailing list
> https://lists.sourceforge.net/lists/listinfo/jflex-users

[jflex-users] ArrayIndexOutOfBounds exception in next_token()

[Dave F. <dav...@or...>]

Hi,

 

I'm try to parse a database specification from oracle. One of the lines reads as follows:

 

 DESCRIPTION {This metric tracks the hourly rate for an employee. This metric provides a snapshot of an employee’s pay rate that will be used in employee information listings. Also, you can analyze hourly rates by jobs, job families, positions, and pay grades to
analyze your pay rate structure. 
} 

 

The parser is set up to ignore everything between the braces.

 

We get an ArrayIndexOutOfBounds exception in next_token() when trying to read the character '€' where zzInput == 226 and zzCMapL has a size of 128. 

 

I would appreciate any insight into this.

 

Thanks, 

 

David Farr

 

Re: [jflex-users] How to remove println()s in JFlex generated class

[Nikhil D. <nik...@gm...>]

Hi,

Do you have the %debug flag in your lexer spec by any chance? It doesn't
print by default, in my recollection.

-Nikhil

On Mon, Mar 16, 2009 at 2:42 PM, James Cant <jc...@he...>wrote:

> Hi,
>
> I hope someone maybe able to help me with a JFlex problem.
>
> I have a string representation of a data structure and hope to tokenize
> it with JFlex and then build the data structure from the token stream.
> I started with the simple example in the distributed examples
> directory.  I got it working and made some of the modifications I'll
> need to do the job.
>
> However, the generated scanner prints out a couple of lines each time
> yylex() is called.  Though this is fine for the example, it is a problem
> for my application.  How can I make these go away?  I didn't find a
> solution in the manual nor did I see where these System.out.println's
> originate in either the skeleton file or my .flex input.
>
> I'd really appreciate your advice (or solution!).  Currently I'm
> removing these lines with a perl script before compiling the JFlex
> generated class; I view this as somewhat inelegant.
> Thanks for your interest.
> jim cant
>
>
> ------------------------------------------------------------------------------
> Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are
> powering Web 2.0 with engaging, cross-platform capabilities. Quickly and
> easily build your RIAs with Flex Builder, the Eclipse(TM)based development
> software that enables intelligent coding and step-through debugging.
> Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com
> --
> jflex-users mailing list
> https://lists.sourceforge.net/lists/listinfo/jflex-users
>

[jflex-users] How to remove println()s in JFlex generated class

[James C. <jc...@he...>]

Hi,

I hope someone maybe able to help me with a JFlex problem.

I have a string representation of a data structure and hope to tokenize 
it with JFlex and then build the data structure from the token stream.  
I started with the simple example in the distributed examples 
directory.  I got it working and made some of the modifications I'll 
need to do the job.

However, the generated scanner prints out a couple of lines each time 
yylex() is called.  Though this is fine for the example, it is a problem 
for my application.  How can I make these go away?  I didn't find a 
solution in the manual nor did I see where these System.out.println's 
originate in either the skeleton file or my .flex input.

I'd really appreciate your advice (or solution!).  Currently I'm 
removing these lines with a perl script before compiling the JFlex 
generated class; I view this as somewhat inelegant.
Thanks for your interest.
jim cant

Re: [jflex-users] A problem with Unicode

[Gerwin K. <ger...@ni...>]

Hi Peter,

I just tried the example you describe and it works for me. I had to take care to 
get the encoding of the input file right, though.

It depends on which platform you are working on and what encoding the input 
(test) data is. It should either be the same as the Java default encoding on 
your platform (you can find our which that is with "jflex --info") or you must 
specify the right encoding when you construct the Reader that reads from the file.

You can see what characters jflex thinks it sees in the spec when you run jflex 
with the option --dump on your spec. The first section is a list of the 
different character classes it recognised and 8220 and 8221 (\u201c+d) should be 
among them. If the character you want is matched by the . rule, try printing out 
the integer value of the char and see if that is indeed 8220 or 8221. If it is 
not, the encoding of the input file is different from what Java expects.

Cheers,
Gerwin

Peter L. Bird wrote:
> I know this must have a simple solution, but i can't figure it out.
> 
> For a normal quoted string (i.e. "This is a string"), i've successfully 
> used the pattern:
> 
>             STRING = [\"][^\"]*[\"] 
> 
> I have text that uses the unicode delimiters* \u201c* and *\u201d. *I 
> thought i could use the same pattern strategy:
> 
>          USTRING = [\u201c][^\u201d]*[\u201d]
> 
> but the pattern never matches.
> 
> I have the %unicode directive set.
> 
> I know the delimiters are read, since my error catching pattern (".") 
> detects them, and i can print out the specific value.  I could certainly 
> process these using exclusive states, but this seems unnecessary.
> 
> What am i doing wrong?
> 
> Peter L. Bird
> pb...@co...
> 
> 
> ------------------------------------------------------------------------
> 
> ------------------------------------------------------------------------------
> Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
> -OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
> -Strategies to boost innovation and cut costs with open source participation
> -Receive a $600 discount off the registration fee with the source code: SFAD
> http://p.sf.net/sfu/XcvMzF8H
> 
> 
> ------------------------------------------------------------------------
> 
> --
> jflex-users mailing list
> https://lists.sourceforge.net/lists/listinfo/jflex-users

Re: [jflex-users] A problem with Unicode

[Lorenzo M. <lo...@re...>]

Hi Peter.
 
I use the following code for STRING:
 
comillas = [\"]
cadena  =   {comillas} .* {comillas}
 
 
Lorenzo Madrid.
lo...@re...
 
 
 
De: Peter L. Bird [mailto:pb...@co...] 
Enviado el: jueves, 26 de febrero de 2009 16:14
Para: jfl...@li...
Asunto: [jflex-users] A problem with Unicode
 
I know this must have a simple solution, but i can't figure it out.

For a normal quoted string (i.e. "This is a string"), i've successfully used
the pattern: 
        STRING = [\"][^\"]*[\"] 
I have text that uses the unicode delimiters \u201c and \u201d. I thought i
could use the same pattern strategy:

         USTRING = [\u201c][^\u201d]*[\u201d]

but the pattern never matches. 

I have the %unicode directive set.

I know the delimiters are read, since my error catching pattern (".")
detects them, and i can print out the specific value.  I could certainly
process these using exclusive states, but this seems unnecessary.

What am i doing wrong?



Peter L. Bird
pb...@co...

[jflex-users] A problem with Unicode

[Peter L. B. <pb...@co...>]

I know this must have a simple solution, but i can't figure it out.

For a normal quoted string (i.e. "This is a string"), i've 
successfully used the pattern:
         STRING = [\"][^\"]*[\"]
I have text that uses the unicode delimiters \u201c and \u201d. I 
thought i could use the same pattern strategy:

         USTRING = [\u201c][^\u201d]*[\u201d]

but the pattern never matches.

I have the %unicode directive set.

I know the delimiters are read, since my error catching pattern (".") 
detects them, and i can print out the specific value.  I could 
certainly process these using exclusive states, but this seems unnecessary.

What am i doing wrong?


Peter L. Bird
pb...@co...