Re: [Scriptonite-dev] some questions (long)

[Etienne M. G. <eg...@j-...>]

"Raif S. Naffah" wrote:
> 
> 1.  UTF-16 / UTF-8
> 
> In "2. Conformance," the ECMA-262 standard states:
> 
> "A conforming implementation of this International standard shall interpret
> characters in conformance with the
> Unicode Standard, Version 2.1 or later, and ISO/IEC 10646-1 with either
> UCS-2 or UTF-16 as..."
> 
> Java understands UTF-8, ie. if we use Reader (or any of its subclasses) we
> would get UTF-8 chars.

Wrong.  UTF-8 is an encoding scheme.  Java characters are simply "char"s.

But: Not all "char"s are valid "java_letter_or_number" (or something like that).  Only
this subset can be used to construct valid identifier.  This is why there is a long subset
definition in the Java grammar.

What we need to know is:
1- Can we represent "Unicode Standard, Version 2.1" characters using "char" (i.e.: is 16
bits enough?)
2- Are there special subset we need to care about (like: valid for identifiers, ...)?.

If the answer to (1) is false, AND the answer to (2) is true, then this complicates the
matter a little.  The solution is then to define chars as follows:

(example)

Helpers
...
  some_multichar_unicode = 0xff34 0x394d;  // a 64 bit unicode char
...
Tokens
...
... = ... some_multichar_unicode ...;
...


Mainly, you simply maintain explicit UTF-16 encoding in the grammar.  You do not build
real 64 (or whatever) bit chars. 

> 
> Since i dont know if we need to convert UTF-8 to UTF-16 (in case it is not
> a subset of the latter), how can we implement the above requirement? of
> course if UTF-8 is a subset of UTF-16 then we need to do nothing, but i
> need a confirmation on that, or a pointer to a document that clearly states it.
> 
> 2. lookahead in lexer.
> 
> On page 19, the standard states:
> 
> "EscapeSequence ::
>          CharacterEscapeSequence
>          0 [lookahead not included in DecimalDigit]
>          HexEscapeSequence
>          UnicodeEscapeSequence"
> 
> how can we enforce this requirement?

Lexer states + custom lexer.

e.g.: You define a bogus token that matches the exception, in the custom lexer, when this
token is matched, you pushback the text into the reader.  (then set token=null)

...
{normal->not_escape_sequence}
   bogus_escape_sequence = 0 decimal_digit;

{normal}
   escape_sequence = ...;

Obviously, token "escape_sequence" is not defined in state "not_escape_sequence".

> 
> 3. Reduction of multiple line_terminators:
> 
> On page 6, the standard states:
> 
> "A MultiLineComment (that is, a comment of the form /*?*/ regardless of
> whether it spans more than one line) is likewise simply discarded if it
> contains no line terminator; but if a MultiLineComment contains one or more
> line terminators, then it is replaced by a single line terminator, which
> becomes part of the stream of input
> elements for the syntactic grammar."
> 
> given that the grammar now looks like the following (lines 53-61)
> 
>     /**
>      *  7.3. Line Terminators. [ECMA-262] p. 12.
>      */
>     lf = 0x000A; // Line Feed
>     cr = 0x000D; // Carriage Return
>     ls = 0x2028; // Line separator
>     ps = 0x2029; // Paragraph separator
> 
>     line_terminator = [lf + [cr + [ls + ps]]];
> 
> should the above statement be interpreted as:
> 
> a. when appropriate, remove all occurences of consecutive line_terminator
> chars and replace with one "canonical line terminator" char? or
> b. when appropriate, remove all occurences of consecutive line_terminator
> chars and replace with the first line_terminator char whatever it is?
> 
> if (a) what should be considered as the "canonical" lt char? a
> scriptonite-specific line_terminator char (chosen among the above list) or
> the Java platform-dependant "line.separator" string?

I would instead define:

line_terminator = (... | multiline_comment)+;

blank = (... | single_line_comment)+;

(Or something like that)

> 
> (SableCC-related; ie. for Etienne)
> 
> 4. lookahead operator
> SableCC 2.13 does not support the "/" lookahead opertor. any ideas if/when
> it will be supported?

Not quite soon.  (Ph.D. thesis has precedence;-)

> 
> 5. Lexer is not generated
> to test the new grammar i commented out the current productions and added
> the following:
> 
>     script = script_token*;
> 
>     script_token =
>          {res} reserved_word
>        | {id}  identifier
>        | {op}  punctuator
>        | {num} numeric_literal
>        | {str} string_literal
>        ;
> 
> with the appropriate supporting productions.
> 
> no Lexer (classes) get generated; why?
> 
> 6. unicode.txt and UnicodeLexer.java
> in sablecc distribution, in the Java1.02 and Java1.1 examples, references
> are made to a 0x001A (Unicode "SUBSTITUTE") character.
> 
> where can i get more information about this character and why it [might]
> appear/s in the input?
> 
> cheers;
> rsn
> 
> _______________________________________________
> Scriptonite-dev mailing list
> Scr...@li...
> http://lists.sourceforge.net/mailman/listinfo/scriptonite-dev

-- 
----------------------------------------------------------------------
Etienne M. Gagnon, M.Sc.                     e-mail: eg...@j-...
Author of SableCC:                 http://www.sable.mcgill.ca/sablecc/
----------------------------------------------------------------------