Re: [Jython-dev] Bug? Jython misinterprets the letter =?iso-8859-1?q?=D6?= (\xd6 in Phyton) within strings as EOF

[<bc...@wo...>]

[Christoph Becker]

>Hi,
>Jython 2.1a1 misinterprets the letter Ö (big O-Umlaut or Unicode u00d6 or \xd6 
>in Phyton) within strings as EOF.
>My JDK ist 1.3.1.
>OS is Windows 98SE
>Local ist German, with characterset  850
>
>This is what Python 2.1 shows if Ö is the only letter in the string
>>>> "Ö"
>'\xd6'
>
>This ist what Jython 2.1a1 shows for the same case
>>>> "Ö"
>Traceback (innermost last):
>  (no code object) at line 0
>  File "<console>", line 2
>SyntaxError: Lexical error at line 2, column 0.  Encountered: <EOF> after : ""
>>>>

The actual error message you get is clearly a bug, but the inability to
use non-ascii characters in dos box is more like a missing feature in
jython.

Windows uses two different character sets, one for the files (cp1205 i
think) and another one for the command prompt (cp850). By default java
only support the file character set and jython only uses the default
encoding from java.

What happens when you enter a big O-Umlaut in a command prompt is this:
1) Java reads it as 0x99.
2) Jython (wrongly) puts it trough the default encoding which
   return unicode 0x2122.
3) This is passed to the javaCC parser which assumes it is only dealing
   with ascii and cuts away the top 8 bits.
4) The result is 0x22 (a double quote) which explains the syntax error.

>I would appreciate a hint on how to bypass or correct this problem if it is not 
>a bug.

You can enter the non-ascii values as hex.


The fix to jython should avoid putting the input through the default
encoding (step 2) until after javaCC have parsed the string. That is not
too hard to do and it will fix the syntax error.

Adding support for multiple encodings is a little harder.

regards,
finn