#123 Support Unicode

[David Costanzo]

FMSLogo was written before Unicode existed and was therefore written as an ANSI application. In the meantime, Windows was rewritten to support Unicode natively and ANSI code pages are supported only for compatibility.

FMSLogo being an ANSI application causes the following problems:

1. It's impossible to use characters that are not within the system code page. For example, it's impossible to, say, use Greek letters on an English build.
2. There is confusing behavior when copying and pasting text that contains Unicode characters.
3. Most libraries don't support ANSI text well. In particular, the wxWidgets has several bugs that when using ANSI strings.
4. It also complicates the translation process, as many translators find it difficult to save files as ANSI text.
5. It complicates the testing process, as testing a non-English build requires changing a system setting and rebooting.
6. It complicates creating a native GNU/Linux project Feature Request #1, as the wxWidgets package in all Linux distributions were compiled without ANSI support.
7. For some languages, notable Chinese and Russian, the editor is much worse than the other languages because of incompatibilies with wxWidgets and the way Chinese ANSI code pages work.
8. It complicates and slows down the build, as each source code is compiled multiple times, one for each language (mostly due to supporting a different editor in #7)
9. There is a risk that Windows will eventually stop supporting ANSI programs, in which case FMSLogo wouldn't work on new versions of Windows.

The most difficult part using Unicode will be maintaining backward compatibility, being able to read old ANSI Logo programs, only saving a file with a BOM if the workspace contains a non-ASCII character, continuing to support ANSI strings in DLLCALL, continuing to send ANSI strings in the network API, updating RAWASCII, CHAR, and ASCII, etc.

I have started work on this and exist it will be released as a new major version, 8.0.0.

[David Costanzo]

I have finished updating FMSLogo to be a Unicode application. It supports the BMP, but will have trouble with code points >65535.

This introduced the following breaking changes:

1. CHAR now wraps at 65536, instead of 256.
2. ASCII and RAWASCII now output numbers from 0-65535 (Unicode BMP). This may result in a different number for most characters outside of the ASCII range.
3. BACKSLASHEDP now outputs TRUE for characters created from code points in the Private Use Area of Unicode, instead of ones in the range of ASCII control characters. Similarly, when RAWASCII inputs such characters it outputs large numbers from the Private Use Area, instead of small ones from the range of ASCII control characters.
4. If a file is opened in text mode and includes a byte order mark at the beginning, then SETWRITEPOS 0 and SETREADPOS 0 now set the position to after the byte order mark, instead of before it.
5. OPENREAD, OPENAPPEND, and OPENUPDATE in text modes now infer the character encoding from the presence of a byte order mark (BOM) character. This results in different characters being read from any text file which has a BOM. OPENUPDATE honors this encoding when writing, resulting in different bytes being written for any file that includes a BOM.
6. When OPENWRITE or OPENAPPEND create text files, they encode all data as UTF-8 and write a BOM at the beginning, whereas previously they created empty files (no BOM) and encoded data in the system default code page.
7. NETCONNECTSENDVALUE and NETACCEPTSENDVALUE now convert from Unicode to the default Windows system code page before transmission. NETCONNECTRECEIVEVALUE and NETACCEPTRECEIVEVALUE do the reverse.
8. The "B" output type for DLLCALL uses the lower byte as each character in the returned word. This may result in differences for any bytes written to these buffers which are outside the ASCII range.
9. Files created with DRIBBLE now begin with a byte order mark and are encoded in UTF-8. Before, there was no BOM and the file was encoded in default system Windows code page.