Thread: [Scriptonite-dev] some questions (long)
Status: Planning
Brought to you by:
egagnon
Menu
▾
▴
scriptonite-dev
|
From: Raif S. N. <ra...@fl...> - 2000-03-27 20:11:51
|
1. UTF-16 / UTF-8
In "2. Conformance," the ECMA-262 standard states:
"A conforming implementation of this International standard shall interpret
characters in conformance with the
Unicode Standard, Version 2.1 or later, and ISO/IEC 10646-1 with either
UCS-2 or UTF-16 as..."
Java understands UTF-8, ie. if we use Reader (or any of its subclasses) we
would get UTF-8 chars.
Since i dont know if we need to convert UTF-8 to UTF-16 (in case it is not
a subset of the latter), how can we implement the above requirement? of
course if UTF-8 is a subset of UTF-16 then we need to do nothing, but i
need a confirmation on that, or a pointer to a document that clearly states it.
2. lookahead in lexer.
On page 19, the standard states:
"EscapeSequence ::
CharacterEscapeSequence
0 [lookahead not included in DecimalDigit]
HexEscapeSequence
UnicodeEscapeSequence"
how can we enforce this requirement?
3. Reduction of multiple line_terminators:
On page 6, the standard states:
"A MultiLineComment (that is, a comment of the form /*
*/ regardless of
whether it spans more than one line) is likewise simply discarded if it
contains no line terminator; but if a MultiLineComment contains one or more
line terminators, then it is replaced by a single line terminator, which
becomes part of the stream of input
elements for the syntactic grammar."
given that the grammar now looks like the following (lines 53-61)
/**
* 7.3. Line Terminators. [ECMA-262] p. 12.
*/
lf = 0x000A; // Line Feed
cr = 0x000D; // Carriage Return
ls = 0x2028; // Line separator
ps = 0x2029; // Paragraph separator
line_terminator = [lf + [cr + [ls + ps]]];
should the above statement be interpreted as:
a. when appropriate, remove all occurences of consecutive line_terminator
chars and replace with one "canonical line terminator" char? or
b. when appropriate, remove all occurences of consecutive line_terminator
chars and replace with the first line_terminator char whatever it is?
if (a) what should be considered as the "canonical" lt char? a
scriptonite-specific line_terminator char (chosen among the above list) or
the Java platform-dependant "line.separator" string?
(SableCC-related; ie. for Etienne)
4. lookahead operator
SableCC 2.13 does not support the "/" lookahead opertor. any ideas if/when
it will be supported?
5. Lexer is not generated
to test the new grammar i commented out the current productions and added
the following:
script = script_token*;
script_token =
{res} reserved_word
| {id} identifier
| {op} punctuator
| {num} numeric_literal
| {str} string_literal
;
with the appropriate supporting productions.
no Lexer (classes) get generated; why?
6. unicode.txt and UnicodeLexer.java
in sablecc distribution, in the Java1.02 and Java1.1 examples, references
are made to a 0x001A (Unicode "SUBSTITUTE") character.
where can i get more information about this character and why it [might]
appear/s in the input?
cheers;
rsn
|
|
From: Raif S. N. <ra...@fl...> - 2000-03-27 21:12:32
|
At 06:05 AM 3/28/00 +1000, Raif S. Naffah wrote:
>...
>5. Lexer is not generated
>to test the new grammar i commented out the current productions and added
>the following:
>
> script = script_token*;
>
> script_token =
> {res} reserved_word
> | {id} identifier
> | {op} punctuator
> | {num} numeric_literal
> | {str} string_literal
> ;
>
>with the appropriate supporting productions.
>
>no Lexer (classes) get generated; why?
pls diregard this point.
cheers;
rsn
|
|
From: Etienne M. G. <eg...@j-...> - 2000-03-27 21:37:58
|
"Raif S. Naffah" wrote:
>
> 1. UTF-16 / UTF-8
>
> In "2. Conformance," the ECMA-262 standard states:
>
> "A conforming implementation of this International standard shall interpret
> characters in conformance with the
> Unicode Standard, Version 2.1 or later, and ISO/IEC 10646-1 with either
> UCS-2 or UTF-16 as..."
>
> Java understands UTF-8, ie. if we use Reader (or any of its subclasses) we
> would get UTF-8 chars.
Wrong. UTF-8 is an encoding scheme. Java characters are simply "char"s.
But: Not all "char"s are valid "java_letter_or_number" (or something like that). Only
this subset can be used to construct valid identifier. This is why there is a long subset
definition in the Java grammar.
What we need to know is:
1- Can we represent "Unicode Standard, Version 2.1" characters using "char" (i.e.: is 16
bits enough?)
2- Are there special subset we need to care about (like: valid for identifiers, ...)?.
If the answer to (1) is false, AND the answer to (2) is true, then this complicates the
matter a little. The solution is then to define chars as follows:
(example)
Helpers
...
some_multichar_unicode = 0xff34 0x394d; // a 64 bit unicode char
...
Tokens
...
... = ... some_multichar_unicode ...;
...
Mainly, you simply maintain explicit UTF-16 encoding in the grammar. You do not build
real 64 (or whatever) bit chars.
>
> Since i dont know if we need to convert UTF-8 to UTF-16 (in case it is not
> a subset of the latter), how can we implement the above requirement? of
> course if UTF-8 is a subset of UTF-16 then we need to do nothing, but i
> need a confirmation on that, or a pointer to a document that clearly states it.
>
> 2. lookahead in lexer.
>
> On page 19, the standard states:
>
> "EscapeSequence ::
> CharacterEscapeSequence
> 0 [lookahead not included in DecimalDigit]
> HexEscapeSequence
> UnicodeEscapeSequence"
>
> how can we enforce this requirement?
Lexer states + custom lexer.
e.g.: You define a bogus token that matches the exception, in the custom lexer, when this
token is matched, you pushback the text into the reader. (then set token=null)
...
{normal->not_escape_sequence}
bogus_escape_sequence = 0 decimal_digit;
{normal}
escape_sequence = ...;
Obviously, token "escape_sequence" is not defined in state "not_escape_sequence".
>
> 3. Reduction of multiple line_terminators:
>
> On page 6, the standard states:
>
> "A MultiLineComment (that is, a comment of the form /*?*/ regardless of
> whether it spans more than one line) is likewise simply discarded if it
> contains no line terminator; but if a MultiLineComment contains one or more
> line terminators, then it is replaced by a single line terminator, which
> becomes part of the stream of input
> elements for the syntactic grammar."
>
> given that the grammar now looks like the following (lines 53-61)
>
> /**
> * 7.3. Line Terminators. [ECMA-262] p. 12.
> */
> lf = 0x000A; // Line Feed
> cr = 0x000D; // Carriage Return
> ls = 0x2028; // Line separator
> ps = 0x2029; // Paragraph separator
>
> line_terminator = [lf + [cr + [ls + ps]]];
>
> should the above statement be interpreted as:
>
> a. when appropriate, remove all occurences of consecutive line_terminator
> chars and replace with one "canonical line terminator" char? or
> b. when appropriate, remove all occurences of consecutive line_terminator
> chars and replace with the first line_terminator char whatever it is?
>
> if (a) what should be considered as the "canonical" lt char? a
> scriptonite-specific line_terminator char (chosen among the above list) or
> the Java platform-dependant "line.separator" string?
I would instead define:
line_terminator = (... | multiline_comment)+;
blank = (... | single_line_comment)+;
(Or something like that)
>
> (SableCC-related; ie. for Etienne)
>
> 4. lookahead operator
> SableCC 2.13 does not support the "/" lookahead opertor. any ideas if/when
> it will be supported?
Not quite soon. (Ph.D. thesis has precedence;-)
>
> 5. Lexer is not generated
> to test the new grammar i commented out the current productions and added
> the following:
>
> script = script_token*;
>
> script_token =
> {res} reserved_word
> | {id} identifier
> | {op} punctuator
> | {num} numeric_literal
> | {str} string_literal
> ;
>
> with the appropriate supporting productions.
>
> no Lexer (classes) get generated; why?
>
> 6. unicode.txt and UnicodeLexer.java
> in sablecc distribution, in the Java1.02 and Java1.1 examples, references
> are made to a 0x001A (Unicode "SUBSTITUTE") character.
>
> where can i get more information about this character and why it [might]
> appear/s in the input?
>
> cheers;
> rsn
>
> _______________________________________________
> Scriptonite-dev mailing list
> Scr...@li...
> http://lists.sourceforge.net/mailman/listinfo/scriptonite-dev
--
----------------------------------------------------------------------
Etienne M. Gagnon, M.Sc. e-mail: eg...@j-...
Author of SableCC: http://www.sable.mcgill.ca/sablecc/
----------------------------------------------------------------------
|
|
From: Raif S. N. <ra...@fl...> - 2000-03-28 20:16:47
|
At 04:31 PM 3/27/00 -0500, Etienne M. Gagnon wrote:
>"Raif S. Naffah" wrote:
> >
> > 1. UTF-16 / UTF-8
> >
> > In "2. Conformance," the ECMA-262 standard states:
> >
> > "A conforming implementation of this International standard shall interpret
> > characters in conformance with the
> > Unicode Standard, Version 2.1 or later, and ISO/IEC 10646-1 with either
> > UCS-2 or UTF-16 as..."
> >
> > Java understands UTF-8, ie. if we use Reader (or any of its subclasses) we
> > would get UTF-8 chars.
>
>Wrong. UTF-8 is an encoding scheme. Java characters are simply "char"s.
???
java chars are 16-bit allright and utf-8 is of course an encoding.
i was referring to java Reader classes reading byte sequences and
translating them into chars. (from the InputStreamReader javadoc:
"An InputStreamReader is a bridge from byte streams to character
streams:
It reads bytes and translates them into characters according to a
specified
character encoding. The encoding that it uses may be specified by
name,
or the platform's default encoding may be accepted. "
Lexers generated by SableCC use a Reader to parse the input stream into the
language tokens. hence, if we know the encoding of the input stream, say
ISO 2022 CN, Chinese, we would then instantiate a (subclass of the) Reader
using the encoding name "ISO2022CN," eg.
isr = InputStreamReader(in, "ISO2022CN");
for the complete list of Java 2 supported encodings, see
<file:///${your-jdk-home}/docs/guide/internat/encoding.doc.html>.
so for me the question remains: do we need to write a ByteToChar converter
for UTF-16?
>But: Not all "char"s are valid "java_letter_or_number" (or something like
>that). Only
>this subset can be used to construct valid identifier. This is why there
>is a long subset
>definition in the Java grammar.
>
>What we need to know is:
>1- Can we represent "Unicode Standard, Version 2.1" characters using
>"char" (i.e.: is 16
>bits enough?)
yes.
>2- Are there special subset we need to care about (like: valid for
>identifiers, ...)?.
yes.
>If the answer to (1) is false, AND the answer to (2) is true, then this
>complicates the
>matter a little. The solution is then to define chars as follows:
>
>(example)
>
>Helpers
>...
> some_multichar_unicode = 0xff34 0x394d; // a 64 bit unicode char
>...
>Tokens
>...
>... = ... some_multichar_unicode ...;
>...
>
>
>Mainly, you simply maintain explicit UTF-16 encoding in the grammar. You
>do not build
>real 64 (or whatever) bit chars.
>
> >
> > Since i dont know if we need to convert UTF-8 to UTF-16 (in case it is not
> > a subset of the latter), how can we implement the above requirement? of
> > course if UTF-8 is a subset of UTF-16 then we need to do nothing, but i
> > need a confirmation on that, or a pointer to a document that clearly
> states it.
> >
> > 2. lookahead in lexer.
> >
> > On page 19, the standard states:
> >
> > "EscapeSequence ::
> > CharacterEscapeSequence
> > 0 [lookahead not included in DecimalDigit]
> > HexEscapeSequence
> > UnicodeEscapeSequence"
> >
> > how can we enforce this requirement?
>
>Lexer states + custom lexer.
>
>e.g.: You define a bogus token that matches the exception, in the custom
>lexer, when this
>token is matched, you pushback the text into the reader. (then set
>token=null)
>
>...
>{normal->not_escape_sequence}
> bogus_escape_sequence = 0 decimal_digit;
>
>{normal}
> escape_sequence = ...;
>
>Obviously, token "escape_sequence" is not defined in state
>"not_escape_sequence".
ok.
> >
> > 3. Reduction of multiple line_terminators:
> >
> > On page 6, the standard states:
> >
> > "A MultiLineComment (that is, a comment of the form /*?*/ regardless of
> > whether it spans more than one line) is likewise simply discarded if it
> > contains no line terminator; but if a MultiLineComment contains one or more
> > line terminators, then it is replaced by a single line terminator, which
> > becomes part of the stream of input
> > elements for the syntactic grammar."
> >
> > given that the grammar now looks like the following (lines 53-61)
> >
> > /**
> > * 7.3. Line Terminators. [ECMA-262] p. 12.
> > */
> > lf = 0x000A; // Line Feed
> > cr = 0x000D; // Carriage Return
> > ls = 0x2028; // Line separator
> > ps = 0x2029; // Paragraph separator
> >
> > line_terminator = [lf + [cr + [ls + ps]]];
> >
> > should the above statement be interpreted as:
> >
> > a. when appropriate, remove all occurences of consecutive line_terminator
> > chars and replace with one "canonical line terminator" char? or
> > b. when appropriate, remove all occurences of consecutive line_terminator
> > chars and replace with the first line_terminator char whatever it is?
> >
> > if (a) what should be considered as the "canonical" lt char? a
> > scriptonite-specific line_terminator char (chosen among the above list) or
> > the Java platform-dependant "line.separator" string?
>
>I would instead define:
>
>line_terminator = (... | multiline_comment)+;
>
>blank = (... | single_line_comment)+;
>
>(Or something like that)
agree. re-reading the standard, i noted this paragraph (near bottom of p. 12):
"Comments behave like white space and are discarded except that, if a
MultiLineComment contains a line terminator character, then the entire
comment is considered to be a LineTerminator for purposes of parsing
by the syntactic grammar."
> >
> > (SableCC-related; ie. for Etienne)
> >
> > 4. lookahead operator
> > SableCC 2.13 does not support the "/" lookahead opertor. any ideas if/when
> > it will be supported?
>
>Not quite soon. (Ph.D. thesis has precedence;-)
>
> > ...
> >
> > 6. unicode.txt and UnicodeLexer.java
> > in sablecc distribution, in the Java1.02 and Java1.1 examples, references
> > are made to a 0x001A (Unicode "SUBSTITUTE") character.
> >
> > where can i get more information about this character and why it [might]
> > appear/s in the input?
any pointers?
> >
> > cheers;
> > rsn
> >
> > _______________________________________________
> > Scriptonite-dev mailing list
> > Scr...@li...
> > http://lists.sourceforge.net/mailman/listinfo/scriptonite-dev
>
>--
>----------------------------------------------------------------------
>Etienne M. Gagnon, M.Sc. e-mail: eg...@j-...
>Author of SableCC: http://www.sable.mcgill.ca/sablecc/
>----------------------------------------------------------------------
|
