Re: [Scriptonite-dev] some questions (long)

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

At 04:31 PM 3/27/00 -0500, Etienne M. Gagnon wrote:
>"Raif S. Naffah" wrote:
> >
> > 1.  UTF-16 / UTF-8
> >
> > In "2. Conformance," the ECMA-262 standard states:
> >
> > "A conforming implementation of this International standard shall interpret
> > characters in conformance with the
> > Unicode Standard, Version 2.1 or later, and ISO/IEC 10646-1 with either
> > UCS-2 or UTF-16 as..."
> >
> > Java understands UTF-8, ie. if we use Reader (or any of its subclasses) we
> > would get UTF-8 chars.
>
>Wrong.  UTF-8 is an encoding scheme.  Java characters are simply "char"s.

???

java chars are 16-bit allright and utf-8 is of course an encoding.

i was referring to java Reader classes reading byte sequences and 
translating them into chars. (from the InputStreamReader javadoc:

         "An InputStreamReader is a bridge from byte streams to character 
streams:
         It reads bytes and translates them into characters according to a 
specified
         character encoding. The encoding that it uses may be specified by 
name,
         or the platform's default encoding may be accepted. "

Lexers generated by SableCC use a Reader to parse the input stream into the 
language tokens. hence, if we know the encoding of the input stream, say 
ISO 2022 CN, Chinese, we would then instantiate a (subclass of the) Reader 
using the encoding name "ISO2022CN," eg.

         isr = InputStreamReader(in, "ISO2022CN");

for the complete list of Java 2 supported encodings, see 
<file:///${your-jdk-home}/docs/guide/internat/encoding.doc.html>.

so for me the question remains: do we need to write a ByteToChar converter 
for UTF-16?

>But: Not all "char"s are valid "java_letter_or_number" (or something like 
>that).  Only
>this subset can be used to construct valid identifier.  This is why there 
>is a long subset
>definition in the Java grammar.
>
>What we need to know is:
>1- Can we represent "Unicode Standard, Version 2.1" characters using 
>"char" (i.e.: is 16
>bits enough?)

yes.

>2- Are there special subset we need to care about (like: valid for 
>identifiers, ...)?.

yes.

>If the answer to (1) is false, AND the answer to (2) is true, then this 
>complicates the
>matter a little.  The solution is then to define chars as follows:
>
>(example)
>
>Helpers
>...
>   some_multichar_unicode = 0xff34 0x394d;  // a 64 bit unicode char
>...
>Tokens
>...
>... = ... some_multichar_unicode ...;
>...
>
>
>Mainly, you simply maintain explicit UTF-16 encoding in the grammar.  You 
>do not build
>real 64 (or whatever) bit chars.
>
> >
> > Since i dont know if we need to convert UTF-8 to UTF-16 (in case it is not
> > a subset of the latter), how can we implement the above requirement? of
> > course if UTF-8 is a subset of UTF-16 then we need to do nothing, but i
> > need a confirmation on that, or a pointer to a document that clearly 
> states it.
> >
> > 2. lookahead in lexer.
> >
> > On page 19, the standard states:
> >
> > "EscapeSequence ::
> >          CharacterEscapeSequence
> >          0 [lookahead not included in DecimalDigit]
> >          HexEscapeSequence
> >          UnicodeEscapeSequence"
> >
> > how can we enforce this requirement?
>
>Lexer states + custom lexer.
>
>e.g.: You define a bogus token that matches the exception, in the custom 
>lexer, when this
>token is matched, you pushback the text into the reader.  (then set 
>token=null)
>
>...
>{normal->not_escape_sequence}
>    bogus_escape_sequence = 0 decimal_digit;
>
>{normal}
>    escape_sequence = ...;
>
>Obviously, token "escape_sequence" is not defined in state 
>"not_escape_sequence".

ok.

> >
> > 3. Reduction of multiple line_terminators:
> >
> > On page 6, the standard states:
> >
> > "A MultiLineComment (that is, a comment of the form /*?*/ regardless of
> > whether it spans more than one line) is likewise simply discarded if it
> > contains no line terminator; but if a MultiLineComment contains one or more
> > line terminators, then it is replaced by a single line terminator, which
> > becomes part of the stream of input
> > elements for the syntactic grammar."
> >
> > given that the grammar now looks like the following (lines 53-61)
> >
> >     /**
> >      *  7.3. Line Terminators. [ECMA-262] p. 12.
> >      */
> >     lf = 0x000A; // Line Feed
> >     cr = 0x000D; // Carriage Return
> >     ls = 0x2028; // Line separator
> >     ps = 0x2029; // Paragraph separator
> >
> >     line_terminator = [lf + [cr + [ls + ps]]];
> >
> > should the above statement be interpreted as:
> >
> > a. when appropriate, remove all occurences of consecutive line_terminator
> > chars and replace with one "canonical line terminator" char? or
> > b. when appropriate, remove all occurences of consecutive line_terminator
> > chars and replace with the first line_terminator char whatever it is?
> >
> > if (a) what should be considered as the "canonical" lt char? a
> > scriptonite-specific line_terminator char (chosen among the above list) or
> > the Java platform-dependant "line.separator" string?
>
>I would instead define:
>
>line_terminator = (... | multiline_comment)+;
>
>blank = (... | single_line_comment)+;
>
>(Or something like that)

agree. re-reading the standard, i noted this paragraph (near bottom of p. 12):

         "Comments behave like white space and are discarded except that, if a
         MultiLineComment contains a line terminator character, then the entire
         comment is considered to be a LineTerminator for purposes of parsing
         by the syntactic grammar."

> >
> > (SableCC-related; ie. for Etienne)
> >
> > 4. lookahead operator
> > SableCC 2.13 does not support the "/" lookahead opertor. any ideas if/when
> > it will be supported?
>
>Not quite soon.  (Ph.D. thesis has precedence;-)
>
> > ...
> >
> > 6. unicode.txt and UnicodeLexer.java
> > in sablecc distribution, in the Java1.02 and Java1.1 examples, references
> > are made to a 0x001A (Unicode "SUBSTITUTE") character.
> >
> > where can i get more information about this character and why it [might]
> > appear/s in the input?

any pointers?

> >
> > cheers;
> > rsn
> >
> > _______________________________________________
> > Scriptonite-dev mailing list
> > Scr...@li...
> > http://lists.sourceforge.net/mailman/listinfo/scriptonite-dev
>
>--
>----------------------------------------------------------------------
>Etienne M. Gagnon, M.Sc.                     e-mail: eg...@j-...
>Author of SableCC:                 http://www.sable.mcgill.ca/sablecc/
>----------------------------------------------------------------------