[BUG] When you open a UTF-16LE encoded file (without any eanding BOM, for example the "Layout.ini" file in "C:\Windows\Prefetch" on Windows XP) it shows only as an 8-bit encoded file with many NULs. There's absolutely no way to convince Launchpad that the file is encoded with UTF-16, as it apparently only "guesses" a wrong encoding type. So:
=> [RFE] Please provide a way to reload the file using another encoding that the currently selected! And don't assume that files are necessarily encoded with UTF-8 or the local ANSI or OEM encoding.
=> [RFE] Maintain a cache for the currently selected encoding that was used when loading the file. Internally, the editor can use any other convenient 8-bit or 16-bit or 32-bit encoding. The file would later be serialized using this encoding again. My opinion is that the editor should always work internally with UTF-16, and should not prohibit entering characters not supported in the currently selected charset, but should be able to detect and show where non-reencodable characters are present (when saving there could be several behavior, but the filr should not be saved silently by default without alerting the user about unsupported characters, instead it should offer a way to either substitute a default character such as SUB or "?", or to use approximations like in ICU charsets or in Windows legacy ANSI filenames).
[RFE] Please provide a full "Visible controls" mode that will show where invisible characters (or characters that can't be distinguished from regular spaces) are present: render them like controls, using some rounded box including the abbreviation. Preferably, use the same abbreviations as used in the Unicode charts. This should support at least:
* distinct whitespaces: NBSP, ZWSP, ZWNBSP (see the "Zs" general category in Unicode)
* additional line/paragraph separators (see the "Zl" and "Zp" general category in Unicode)
* joiners/disjoiners: CGJ, ZWJ, ZWNJ.
* text directionality controls: LRE/RLE/LRM/RLM.
[RFE] Optionally, and separately, a mode for displaying:
* combining characters or combining sequences separately without rendering the combination (even if they are canonically equivalent).
* detecting ompatibility characters that have a canonical equivalent (so that they can be substituted), such as the Angström sign (that a user would want to canonicalize to the Latin capital A with latin)
[RFE] Include also the support for the four Unicode normalizations (NFC, NFD, NFKC, NFKD); but don't force the canonicalization when reading the file, this should be a script to run.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Perhaps a better approach might be to try to find out why it guesses wrongly, and try to find out what to do to get it to guess correctly. Perhaps it's possible to modify the file slightly before opening it?
I have to admit that I know very little about this type of file. Thanks for providing an example - I've seen complaints about this before but had no sample to try it on.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
[BUG] When you open a UTF-16LE encoded file (without any eanding BOM, for example the "Layout.ini" file in "C:\Windows\Prefetch" on Windows XP) it shows only as an 8-bit encoded file with many NULs. There's absolutely no way to convince Launchpad that the file is encoded with UTF-16, as it apparently only "guesses" a wrong encoding type. So:
=> [RFE] Please provide a way to reload the file using another encoding that the currently selected! And don't assume that files are necessarily encoded with UTF-8 or the local ANSI or OEM encoding.
=> [RFE] Maintain a cache for the currently selected encoding that was used when loading the file. Internally, the editor can use any other convenient 8-bit or 16-bit or 32-bit encoding. The file would later be serialized using this encoding again. My opinion is that the editor should always work internally with UTF-16, and should not prohibit entering characters not supported in the currently selected charset, but should be able to detect and show where non-reencodable characters are present (when saving there could be several behavior, but the filr should not be saved silently by default without alerting the user about unsupported characters, instead it should offer a way to either substitute a default character such as SUB or "?", or to use approximations like in ICU charsets or in Windows legacy ANSI filenames).
[RFE] Please provide a full "Visible controls" mode that will show where invisible characters (or characters that can't be distinguished from regular spaces) are present: render them like controls, using some rounded box including the abbreviation. Preferably, use the same abbreviations as used in the Unicode charts. This should support at least:
* distinct whitespaces: NBSP, ZWSP, ZWNBSP (see the "Zs" general category in Unicode)
* additional line/paragraph separators (see the "Zl" and "Zp" general category in Unicode)
* joiners/disjoiners: CGJ, ZWJ, ZWNJ.
* text directionality controls: LRE/RLE/LRM/RLM.
[RFE] Optionally, and separately, a mode for displaying:
* combining characters or combining sequences separately without rendering the combination (even if they are canonically equivalent).
* detecting ompatibility characters that have a canonical equivalent (so that they can be substituted), such as the Angström sign (that a user would want to canonicalize to the Latin capital A with latin)
[RFE] Include also the support for the four Unicode normalizations (NFC, NFD, NFKC, NFKD); but don't force the canonicalization when reading the file, this should be a script to run.
Perhaps a better approach might be to try to find out why it guesses wrongly, and try to find out what to do to get it to guess correctly. Perhaps it's possible to modify the file slightly before opening it?
I have to admit that I know very little about this type of file. Thanks for providing an example - I've seen complaints about this before but had no sample to try it on.
Well, you better start working on it right away, 'cause you got a lot to do!