[Pythoncard-users] Fwd: PythonCard Unicode question

[Kevin A. <al...@se...>]

For those of you that have had problems with 8-bit ascii or unicode characters with PythonCard, the following will probably be of interest to you. The error below concerned an example program in the book "Hello World!: Computer Programming for Kids and Other Beginners" by Warren D. Sande and Carter Sande. The problem occurs when using something other than 7-bit ASCII characters in a data file with the hangman example program.

http://www.manning.com/sande/

The thread on the forum for the book is here:

http://www.manning-sandbox.com/thread.jspa?threadID=40350&tstart=0

The problem wasn't with PythonCard per se, but rather that by default Python 2.x uses ASCII to read text files.

When you are dealing with something other than 7-bit ASCII you need to do some extra work for built-ins like open().

http://docs.python.org/howto/unicode.html

http://www.python.org/dev/peps/pep-0263/

There are many 8-bit character encodings that vary across operating systems and languages, but Unicode UTF-8 is a standard for 8-bit encoding that will generally work for languages using the latin (roman) alphabet. So in the case below, to cover umlauts in the text file I suggested using "utf-8" with the codecs.open method. Of course, you need to make sure that your text editor or program creating the data file generates utf-8 text otherwise the encoding needs to be changed.

There are a numerous places in PythonCard that could probably be changed to work better with non-ascii characters. In particular the readAndEvalFile function in util.py which is used to read resources and configuration files should probably be changed to use codecs.open() instead. I found possible places to update in PythonCard by using the findfiles tool PythonCard/tools/findfiles/findfiles.py to search for relevant regular expressions like open\( in case you want to look through the code yourself.

I'm not an expert on text encoding issues, so anyone that has more experience in these matters is welcome to make suggestions for code changes in the PythonCard framework and samples that would allow it to work more seamlessly with non-ascii text files, including resources used by PythonCard programs. Typically, I would expect you to see errors when using non 7-bit ASCII characters in menus, fields, or any other data associated with resource files based on the problem brought up by Warren and his reader.

Note that for testing purposes, I'm now using Python 2.7.x and a unicode build of wxPython on Mac OS X and Windows. While Python 3.x uses Unicode by default, wxPython doesn't support Python 3.x yet, thus PythonCard won't work under Python 3.x and I haven't switched to Python 3.x for anything I do with Python.

Thanks,

ka


Begin forwarded message:

> From: Kevin Altis <al...@se...>
> Date: November 5, 2010 10:01:44 PM PDT
> To: HelloWorldBook Help <cp4...@ya...>
> Subject: Re: PythonCard Unicode question
> 
> Hi,
> I think I know the issue, but would like to get some confirmation and then reply to the list once we are sure of the solution.
> 
> The line you need to add is:
> 
> import codecs
> 
> Then change your f=open("words.txt", 'r') line to:
> 
> f = codecs.open("words.txt", encoding="utf-8")
> 
> The relevant documentation files for Python 2.6/2.7 are:
> 
> http://docs.python.org/howto/unicode.html
> 
> and
> 
> http://www.python.org/dev/peps/pep-0263/
> 
> The PEP is only relevant if you are trying to use non-ascii in the source file, but is good background anyway. For example, I added a test line that printed out some Unicode characters I embedded in the file, which required the encoding line, but otherwise shouldn't be needed.
> 
> I tried your source files with Python 2.7 on Mac OS 10.6.4 (Snow Leopard) using the Python 2.7 build from python.org and the Unicode wxPython build for Python 2.7. I used the supplied words.txt file and one that I created in UTF-8 text format utilizing umlauts and other special characters.
> 
> This is actually a fairly complicated issue, but assuming you are using a recent Python other than Python 3.0 and a unicode build of wxPython I think this solves the problem.
> 
> Let me know if it works and I'll follow-up to the list.
> 
> BTW, I would like to see the PythonCard references in the book sometime.
> 
> Thanks,
> 
> ka
> 
> On Nov 5, 2010, at 2:08 PM, HelloWorldBook Help wrote:
> 
>> Kevin,
>>  
>> We introduce PythonCard in our kids/beginners programming book.  We use it to make a Hangman game (among other things).  The list of words from which the puzzle is randomly chosen is stored in a text file.  A reader posted the following question on our message board:
>> 
>> Subject:  How can I get hangman to accept umlauts, such as 'ä' etc?
>> 
>> Hi,
>> 
>> To test whether I could use German words in hangman, I replaced the original file words.txt of hangman with a words.txt file containing only words with umlauts, such as 'ä', 'ö', 'ü'. When running hangman.py and entering, say 'ä', I get this error message:
>> 
>> Traceback (most recent call last):
>> File "/usr/lib/pymodules/python2.6/PythonCard/widget.py", line 408, in _dispatch
>> handler(background, aWxEvent)
>> File "hangman.py", line 109, in on_btnGuessLetter_mouseClick
>> if result.text in self.currentword:
>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)
>> 
>> I am using Hello World both with Windows XP and with Ubuntu-Linux. The error messages are the same.
>> 
>> How can I get hangman accepting umlauts and 'ß'? I would like to use German words, and they contain a lot of umlauts - and ß.
>> 
>> Can you help answer this?  You can find the code for the Hangman game here:
>> http://www.manning.com/sande/sourcecode/All_Files_By_Chapter/hw_ch22_code/
>> (hangman.py and hangman.rsrc.py)
>>  
>> Thanks,
>> Warren Sande
>>  
>> 
>> 
>