[Scriptonite-dev] some questions (long)

[Raif S. N. <ra...@fl...>]

1.  UTF-16 / UTF-8

In "2. Conformance," the ECMA-262 standard states:

"A conforming implementation of this International standard shall interpret 
characters in conformance with the
Unicode Standard, Version 2.1 or later, and ISO/IEC 10646-1 with either 
UCS-2 or UTF-16 as..."

Java understands UTF-8, ie. if we use Reader (or any of its subclasses) we 
would get UTF-8 chars.

Since i dont know if we need to convert UTF-8 to UTF-16 (in case it is not 
a subset of the latter), how can we implement the above requirement? of 
course if UTF-8 is a subset of UTF-16 then we need to do nothing, but i 
need a confirmation on that, or a pointer to a document that clearly states it.


2. lookahead in lexer.

On page 19, the standard states:

"EscapeSequence ::
         CharacterEscapeSequence
         0 [lookahead not included in DecimalDigit]
         HexEscapeSequence
         UnicodeEscapeSequence"

how can we enforce this requirement?


3. Reduction of multiple line_terminators:

On page 6, the standard states:

"A MultiLineComment (that is, a comment of the form /*…*/ regardless of 
whether it spans more than one line) is likewise simply discarded if it 
contains no line terminator; but if a MultiLineComment contains one or more 
line terminators, then it is replaced by a single line terminator, which 
becomes part of the stream of input
elements for the syntactic grammar."

given that the grammar now looks like the following (lines 53-61)

    /**
     *  7.3. Line Terminators. [ECMA-262] p. 12.
     */
    lf = 0x000A; // Line Feed
    cr = 0x000D; // Carriage Return
    ls = 0x2028; // Line separator
    ps = 0x2029; // Paragraph separator

    line_terminator = [lf + [cr + [ls + ps]]];

should the above statement be interpreted as:

a. when appropriate, remove all occurences of consecutive line_terminator 
chars and replace with one "canonical line terminator" char? or
b. when appropriate, remove all occurences of consecutive line_terminator 
chars and replace with the first line_terminator char whatever it is?

if (a) what should be considered as the "canonical" lt char? a 
scriptonite-specific line_terminator char (chosen among the above list) or 
the Java platform-dependant "line.separator" string?


(SableCC-related; ie. for Etienne)

4. lookahead operator
SableCC 2.13 does not support the "/" lookahead opertor. any ideas if/when 
it will be supported?


5. Lexer is not generated
to test the new grammar i commented out the current productions and added 
the following:

    script = script_token*;

    script_token =
         {res} reserved_word
       | {id}  identifier
       | {op}  punctuator
       | {num} numeric_literal
       | {str} string_literal
       ;

with the appropriate supporting productions.

no Lexer (classes) get generated; why?


6. unicode.txt and UnicodeLexer.java
in sablecc distribution, in the Java1.02 and Java1.1 examples, references 
are made to a 0x001A (Unicode "SUBSTITUTE") character.

where can i get more information about this character and why it [might] 
appear/s in the input?


cheers;
rsn