#727 (Partial) Unicode support for wgnuplot terminal window
The wgnuplot terminal window currently accepts input from the keyboard and clipboard only in the "ANSI" single-byte code page. The attached patch implements (limited) Unicode support for keyboard and clipboard input. Note that most terminals, including the windows terminal, already accept UTF8 encoded strings, but you previously had to load commands from a file.
The patch makes wgnuplot accept UTF16 encoded keyboard input (Windows default). Internally, this input is converted to whatever the currently set encoding is. It uses UTF8 if the encoding is set to utf8. wgnuplot converts all screen output to UTF16 again and then uses Windows's wide-character API to display it.
Limitations: The current code still assumes number of wide-characters ~ length of output. This means that only the UCS-2 character set and half-width characters are supported. Full-width characters should display correctly, but cursor navigation is off.
Also, the C library does not know about the encoding and hence e.g. file names which contain non-ASCII characters may not work correctly. Not sure how to fix this properly.
Note that this patch does not affect console mode gnuplot on Windows.
Related
A remark about SJIS encoding: in principle this encoding is supported for input. But the built-in readline does not currently handle it and the display routine does not handle SJIS either. I will try to fix at least the latter.
Some comments/questions:
I don't understand how this approach is affected by the encoding currently in use on the user's desktop (as distinct from "set encoding" in gnuplot). Does this fix Bug #1558? If it doesn't then it will probably fail for odd corner cases with other encodings as well.
I also don't follow your remark about text width. If text is converted to UTF-8 on input, gnuplot's own routines will properly count the text width in terms of displayed characters rather than the number of bytes. So the text layout should all "just work" at that point, even if the text is converted back to some other encoding on output. Or are you just referring to the text echo in readline()? But that also works fine for UTF-8, so I guess I'm missing something.
Would it be possible to do the conversion in a separate filter program such that input goes in a piped sequence
??
For instance here is a MSDN document that seems to imply that you could process the input as a stream of key events and then have a lookup table to convert key events to UTF-8 characters:
https://msdn.microsoft.com/en-us/library/windows/desktop/gg153546(v=vs.85).aspx
For that matter, isn't there some equivalent of ibus/SCIM for Windows? Is that what TSF (Text Services Framework) is for?
Ethan, thanks for the feedback. Note that we already discussed this approach together with Mojca via email in 2011. The idea is to let the user choose the internal string encoding via
set encoding.The input is "magically" unaffected. Windows provides WM_CHAR messages to GUI programs which contain Unicode in the form of UTF-16, so no codepage involved.
The only point where I currently do expect problems is
fopenwith non-ASCII characters in the file name. The fix for this is also straight forward: use_wfopeninstead. But that is not yet in the patch. Note that Windows C runtimes apparently do not support setting thelocaleto UTF-8 encoding.Ouch. Lost track of #1558, which is trivial to fix.Will do so asap.
You are right. gnuplot's internals handle UTF-8 just fine. It's the GUI output code (wtext.c, screenbuf.c) that makes that assumption about the UTF16 encoded text: it doesn't handle surrogates nor full-width characters. These are issues which could be addressed later.
No, not for
wgnuplot. For console mode gnuplot that might be possible, but the problems are actually on the output side. Also here using the UTF-16 code already provided byReadConsoleInputto get UTF-8 input should be easy enough.That's about what the patch does: The "filter" is a few lines addition to the WM_CHAR message treatment to convert UTF-16 to UTF-8 and then send that to gnuplot's internal readline.
No idea really. The Microsoft Input Method Editor, IME, maybe?
Last edit: Bastian Märkisch 2016-03-26
It turns out that that statement wasn't the whole truth. Windows' C runtime support for UTF-8 is very limited. We cannot set the locale to use UTF-8 encoding and thus will have to use our own translation between UTF-8 and UTF-16 wherever that matters:
fopen,system,popen, drag'n drop, copy-paste, dialog windows, filename tab-completion, pause command. The attached patch handles all these cases. In addition the display code now also handles S-JIS. The additional changes are pretty straight forward, but nevertheless quite numerous. diffstat's summary: 18 files changed, 1232 insertions(+), 1025 deletions(-)Potential Changelog entry:
1) I am a bit uneasy about tying this to "set encoding". Maybe it's OK, but the original idea of that command was to choose the output encoding, not the input encoding.
2) The terminology "half-width / full-width" characters is extremely confusing because it means two quite different things. I realize this confusion is the fault of some committee, but the fact remains that I can't tell from the text here whether the patch supports normal CJK unicode characters or not. These are "full width" in the sense of character layout, but the majority of them require only 2 bytes in UTF_16. "half width" kana characters also require only 2 bytes. If I understand correctly (not 100% sure) the actual limitation is to Unicode plane 0, which includes both half-width and full-width characters in the CJK sense.
3) How does this patch affect people who normally use SJIS (or any other non-UTF8 encoding)?
1) I am certainly open to make this depend on
inputencodingor similar. The reason why I choseencodingwas that already now the internal readline code which is used on Windows has that behaviour. Until now this effectively "breaks" input whenset encoding utf8is used. So that should be changed, too. On Unix one would probably make "inputencoding" depend on the locale's encoding, but on Windows UTF-8 locales are not allowed.2) Maybe my use of terms is wrong. What I mean is that for all characters with
wcwidth(c) >1cursor movement and editing is not yet working correctly.That hasn't changed.The terminal text backend currently assumes width-on-screen(character) == const for all characters, where only double-byte characters are allowed (no surrogates). This stems from the very straightforward change from single-byte-encoded char to WCHAR. U+0000 to U+D7FF and U+E000 to U+FFFF should display fine - given the current font supports them.3) If the user hasn't explicitely specified an
encoding,WinGetCodepage()returnsCP_ACP, which is the default "Ansi" codepage for that process. The internal conversion process is thus transparent to the user, i.e. there should be no change in behaviour.The only double-byte encoding currently supported by gnuplot is S-JIS. Support for this is limited even now because editing is broken for two reasons:
a) the builtin readline does not handle double-byte characters
b) the text window backend assumes one byte per character
c) the text window assumes width of character == const
With the patch the situation is only slightly improved, as only b) is addressed. Since the work has been done for the built-in readline and UTF-8 already, it shouldn't be too hard to implement a). Fixing c) might be a bit tricky.
It is probably best to upload a binary to SF for testing.
1) I don't think unix is sensitive to the input encoding, so it doesn't need a separate command to set it. Under the "stream of bytes" model it doesn't matter what the bytes mean. It only matters that we can recognize the pre-defined keywords. Byte strings are passed through unchanged to the output. What the "set encoding" command does is to label that output for terminal types that care. Typically this is so that the terminal/browser/printer/etc can select a font with a compatible encoding. The core gnuplot code does not care. The command "set encoding locale" labels the output with whatever the input locale was. "set encoding utf8" (or "set encoding locale" for a utf-8 locale) has one special side effect. It enables the string width calculation code that accounts for the case where multiple input bytes produce only one (possibly wider-than-usual) glyph on output. This does not change the output byte stream, but it does change how much space on the page is reserved to hold it. This is, I guess, related to your point (2) about character widths.
2) This must be Windows-specific? The wcwidth() function is not relevant on linux because we aren't storing or manipulating (wchar_t) entities. I guess I'm still confused as to exactly what your patch does. If it is converting to an internal utf-8 byte stream then the issue of what wcwidth() would have returned for a different internal storage format is moot.
"The terminal text backend currently assumes width-on-screen(character) == const for all characters, U+0000 to U+D7FF and U+E000 to U+FFFF should display fine" Those codepoint ranges describe almost the entire code plane 0, and it is definitely not true that all characters in that range have the same width on-screen. Most CJK characters ( U+3000 to U+9FFF plus some others) are wider. This is approximated during page layout using estimate.trm -> ENHest_writec(). It is approximated in the builtin readline code by the mbwidth() routine. A better job can be done by avoiding backspace altogether and instead redrawing the entire input line up to the previous character. I thought we were doing that already at least for some editing operations but it's been a while since I looked at that code.
"only double-byte characters are allowed (no surrogates)" I have not encountered the term "surrogates" before this. The wikipedia page for utf-16 leads me to believe it is a terminology only useful with regard to utf-16 access to unicode code planes above plane 0 (the basic multilingual plane BMP). I suggest removing all mention of surrogates and just stating that the patch only handles code plane 0.
3) I was under the impression that SJIS encoding is working currently with the builtin readline. Certainly there are many Japanese gnuplot users and they must be entering text somehow! I suggest asking Tatsuro Matsuoka or Shigeharu Takeno, since I have never managed to get a working SJIS environment set up for testing input. SJIS is not a exactly a two-byte encoding. It is variable-length (either 1 or 2 bytes), but unlike utf-8 you cannot tell by looking a some single byte whether it is or is not part of a multibyte sequence and if so whether it is the 1st or 2nd byte.
Anyhow I think it's great that the patch fills a gap in gnuplot capabilities on Windows. I remain confused as to whether CJK characters are handled correctly, but I guess we'll hear very quickly from Japanese testers if they aren't.
Attached a screenshot with strings taken from "utf8.dem".
An updated binary is now at https://sourceforge.net/projects/gnuplot/upload/gnuplot/testing/
It includes S-JIS support in gnuplot's own readline [#729], and basic support for wide characters (S-JIS & UTF-8): editing on the command line now works with e.g. wide CJK characters as long as the line does not wrap.
Related
Patches:
#729I have downloaded gp510-20160401-win64-mingw-testing-unicode.zip and do the quick test.
I have copied Japanese characters (UTF-8N) and pasted to the wgnuplot command window.
I also have do the same on my build 64 binary today.
These results are attached the following posts.
BTW, the ChangeLog date of your binary is 2014-06-09.
Why it is so old?
Hi Tatsuro, thanks for testing. With the new binary you should set
set encoding utf8orset encoding sjisfirst. Without it it actually should work for sjis characters on your Japanese system, but apparently it doesn't. I'll look into it why.Your binary
My binary
Unexpectedly on unpatched binary pasted Japanese characters are represented correctly.
I suspect that the clipboard on Japanese version windows do something for character coding in copy and paste. (But I do have no evidence.)
Hello Bastian.
I cannot see your reply here.
I have tried 'set encoding utf8'. A resuit is attached.
Turns out that there was an important change missing. The text window was created using the 8bit API and hence WM_CHAR messages were 8bit only. Why it still worked on my system I do not know. That should be fixed now.
Today's patch goes a significant step further: wgnuplot is built as a true Windows Unicode application. While this is not a major change in functionality or code structure, it involves numerous changes to text types, constants and functions. Most of this actually was search-and-replace. Also indentation was changed in various places. Diffstat:
32 files changed, 3176 insertions(+), 2527 deletions(-)A new binary is located in the
testingdirectory.1) Was the intent to replace fopen() with gp_fopen() everywhere? If so I think it would be cleaner to #define fopen() as a macro on Windows and leave the core source files unchanged. If not, then some sort of comment or explanation is needed so that in the future people will know which one to usein new code.
2) This question may not make any sense, but here goes...
The isident() macro defined near the top of scanner.c allows 8-bit characters to appear in variable names. For example, in gnuplot 5 you can say
σ = 1.0; µ = 10.0
plot f(σ,µ)
So far as I know the macro currently works regardless of the encoding. Does it still work on Windows after your patch?
1) Yes. You are right a
#definewould be the less intrusive solution.2) Yes. Still works. Only that previously you could not have entered that e.g. in the codepage 1252 which is used in Germany.
Re-defining a C library function turns out to be tricky, but doable. Revised patch attached which includes a few more fixes and is slightly cleaned-up:
29 files changed, 3022 insertions(+), 2568 deletions(-)
New binary available at https://sourceforge.net/projects/gnuplot/files/gnuplot/testing/
Sorry for late testing. This time, perhaps resllts are what you expect. See attachment file.
The major part of this patch (and functionality) is now in CVS. At this time wgnuplot is still using the byte oriented "ANSI" Windows API by default, but it uses the "Wide" UTF16 API where required to handle all encodings supported by gnuplot for input (keyboard, copy-paste, etc.) and screen output. This is independent of the locale or codepage used by Windows. The code also handles UTF-8.
The transition to a full Windows-Unicode application (which defaults to the UTF16 API set) is now pretty much straight-forward and will come at a later stage. Yet this will touch many lines of code in the Windows part.
I am investigating two ways to let console mode gnuplot hanlde Unicode input, too. For one, we could use WinEditLine which delivers an UTF-8 encoded byte stream, or we could use the UTF16 Windows Console API directly from the built-in readline. In both cases we would have to introduce a similar redirection layer for C library calls like printf etc. as is already done for wgnuplot.
Your work is quite important for users who uses multibytye characters under utf8 encoding.
I do not have enough knowledge which selection is better for console mode of gnuplot (WinEditLine or UTF16 Windows Console API). I will be a tester on your future works.