#9 codec error from string constant

[Glenn Linderman]

I didn't see anything in the documentation about what codec is used by pyexpander. I would have hoped that expander3 would default to Unicode. My UTF-8 source file contains a Chinese character in a string constant, and I got a UnicodeDecodeError. Granted the Windows itself doesn't handle UTF-8 very well in all circumstances, but Python3 generally does.

Here's a stack trace:
Traceback (most recent call last):
File "C:\Program Files\Python36\Scripts\expander3.py", line 133, in <module>
main()
File "C:\Program Files\Python36\Scripts\expander3.py", line 129, in main
process_files(options,args)
File "C:\Program Files\Python36\Scripts\expander3.py", line 50, in process_files
options.include)
File "c:\program files\python36\lib\site-packages\pyexpander\lib.py", line 1794, in expandFile
return processToPrint(parseFile(filename, no_stdin_warning), filename,
File "c:\program files\python36\lib\site-packages\pyexpander\lib.py", line 129, in parseFile
st= f.read()
File "c:\program files\python36\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 8426: character maps to <undefined>

[Glenn Linderman]

stdout was redirected to a file.
I'm just experimenting with pyexpander... it looks nice, but it was hard to determine where some errors occurred... I had to "bisect" search the file to discover where the problem was. I guess incremental development of macros is necessary so errors would be isolated to recent changes. But this one, while it points right to the error character, gives no clue as to how to fix it.

[Glenn Linderman]

Well, due to the difficulty of finding bugs, because the reports don't tell which macro is failing (in all cases), and the unknown character encoding used that caused this problem, and the limitation of having to have balanced () within macro invocations, I decided to write my own. Trying to use yours gave me some good ideas, some from your documented features, and some from the undocumented issues I encountered and listed. I've got everything working now, so no need to fix this bug except for other people.

[Glenn Linderman]

As requested per email (that must not have come via the ticket system), here is a command:

expander3.py balderdash.txt

And attached file. I had moved on from pyexpander as mentioned above, and thought that any Chinese character would reproduce the problem, but maybe it has to be the right one, or in the right context, but I still recalled which line I'd eventually tracked this down to, and was able to recreate it by pasting that line into the attached text file.