Re: [TCLCORE] Customers with "Bengali" as keyboard input language

[Jeff H. <je...@ac...>]

I would actually lean towards believing there are a few ascii commands in
the Tcl/Tk code that could be permanently set over to wide, causing issues
like this.  I thought about doing this a few years ago (when NT became the
solid supported base), but never finished the work.  I believe there is no
more need for the A/W command jumping on Windows.  That can all be
simplified to pure use of W functions.

Jeff

On Wed, Sep 17, 2014 at 7:44 AM, Robert Neumann <RNe...@vi...> wrote:

>  Hello sebres,
>
>
>
> thank you for your answer.
>
>
>
> I tested on Windows 8 where it is easy to install multiple keyboard input
> languages and switch between them on the fly using “Win+Space”. Windows
> will sent a WM_INPUTLANGCHANGE message to an application when the keyboard
> is switched to a different language. The language of Windows itself (user
> interface, message boxes etc.) remains German all the time. Only the
> keyboard is switched to Bengali.
>
>
>
> If I understand you right, you doubt that TK is translating input using
> codepages the way I presume and my system is simply setup in a way that
> does not work for Bengali. But Bengali **does** work on my machine and
> that of our customer in every textbox (Notepad, Notepad++, Word etc.)
> except for TK text boxes and it seems to be because of the code path
> processing keyboard input being *not fully Unicode*.
>
>
>
> And this already starts on the Windows side before the keycode even
> reaches TK:
>
> It seems that in first place Windows is applying codepages because of TK
> creating its windows in a way which makes them **ANSI** for the windows
> API.
>
> What I could determine with the debugger and using Spy++ is that TK is in
> fact receiving only 8-bit character codes in WM_CHAR message, as an ANSI
> window would receive.
>
>
>
> This code is not always the same for the “A” key, but it is always 8-bit:
>
> -          With German keyboard language the code sent for “A” is 0x41
>
> -          With Arabian keyboard language the code sent for “A” is 0xF6
>
> -          With Bengali keyboard language the code sent for **every**
> character is 0x3F (=”?”)
>
>  TK will then translate those codes to UTF-8, for which it needs a
> codepage again – and it has one for Arabian and Russian. But it hasn’t for
> Bengali and if it always receives 0x3F, there is nothing to be done anyway.
>
>
>
> From my current level of research this happens because Windows uses
> codepages to convert keyboard input to ANSI codes if the receiver is an
> ANSI window. Windows has no codepage for Bengali so it sends “?” (
> http://www.unicode.org/mail-arch/unicode-ml/y2001-m04/0148.html).
>
>
>
> That TK creates the window using CreateWindowW() is not enough to make it
> receive Unicode messages:
>
>
>
> Only if you create a window using CreateWindowW() **and** also use
> GetMessageW() and DispatchMessageW() the character codes in WM_CHAR become
> real Unicode:
>
> -          With German keyboard language the code sent for “A” will be
> 0x41
>
> -          With Arabian keyboard language the code sent for “A” will be
> 0x650
>
> -          With Bengali keyboard language the code sent for “A” will be
> 0x993
>
>
>
> This is reproducible on every Windows 7 and 8 machine by installing
> additional keyboard languages (you don’t need to change the language of
> Windows itself).
>
>
> Best regards,
>
> Robert.
>
>
>
> *Von:* Dipl. Ing. Sergey G. Brester [mailto:se...@us...]
> *Gesendet:* Mittwoch, 17. September 2014 16:15
> *An:* Robert Neumann
> *Cc:* tcl...@li...
> *Betreff:* Re: [TCLCORE] Customers with "Bengali" as keyboard input
> language
>
>
>
> @all: sorry for german letter, just to explain easier ...
>
> ---
>
> Hallo Herr Neumann,
>
> it seems that all input is translated using codepages.
>
> Ich denke nicht, dass das richtig ist - denn wäre es so, sollte dann mein
> folgendes Test scheitern:
>
> Getestet habe ich mit Standart-Demo-Widget "Entries without scrollbars"
> aus TK-Demos von Activestate-Standardinstallation in tk8.5 und tk8.6
> (leider habe ich 8.4 momentan nicht zur Hand).
> Ich habe da in ein leereres Editbox ein zweisprachiges Text eingegeben, un
> zwar habe ich German und Russian keyboard layout parallel getestet. Mein
> System-Encoding ist German (cp1252). So ich könnte z.B. mit dem Umschalten
> folgenden Satz eintippen:
> *Привет - äöüß*
>
> Hätte tk (wie Sie es vermuten) es zwischenkonvertiert, so hätte ich dann
> folgendes eingetippt:
>
> Für system encoding cp1252:
> *Ïðèâåò - äöüß*
>
> Oder für system encoding cp1251:
> *Привет - дцьЯ *
>
> Ich denke es liegt an irgendwas anderem - sei es Treiber oder "Microsoft
> Indic Language Input Tool" oder etwas noch.
>
> Haben Sie es mit Activestate-Standardinstallation schon mal ausprobiert?
> Denn wenn es da gehen wird - könnte es dann an Ihrem Compiler bzw.
> Compiler-Einstellungen liegen.
>
> Leider habe ich keine Bengali/Windows zur Verfügung.
> Könnten Sie die Problem-Umgebung etwas präziser beschreiben:
> Windows-Version, System-Encoding, etc.
> Oder könnte man evtl. das "Kundensystem" als Virtuelle Maschine (VM)
> einrichten?
>
> --
>
> MFG,
>
> sebres.
>
>
>
> Am 17.09.2014 12:52, schrieb Robert Neumann:
>
>  Hello everyone,
>
>
>
> we have been using TCL/TK for the user interface of our modelling tool
> from the very beginning 16 years ago:
>
> http://www.vizrt.com/products/viz_artist/
>
> This is the worldwide most used application for creation of realtime
> broadcast graphics.
>
>
>
> For our first customers in Asia we implemented a custom IME editor, but
> the custom editor was optional and customers worldwide have long stopped
> using it, since TK textbox has sufficient unicode handling and Windows
> provides input methods for every language.
>
>
>
> This works for nearly every language (including Thai, Arabian etc.) but we
> have some issues with customers using some indic languages like “Bengali”
> as the keyboard input language. Their input arrives as “???” in the TK text
> widget (that’s even true for the TCL console itself). You can paste Bengali
> characters from other applications though and they will be displayed
> correctly, so it seems to be a problem with processing of keyboard input.
>
>
>
> I have dug into the TK source code (we are still working on 8.4) a bit and
> it seems that all input is translated using codepages. But there are no
> Windows codepages for Bengali (
> http://www.unicode.org/mail-arch/unicode-ml/y2001-m04/0148.html).
>
>
>
> If you create a window as a unicode window (CreateWindowW) **and** use
> the wide character variants of **all** functions driving the message loop
> you will receive unicode WM_CHAR events, which already contain Unicode
> character codes and don’t need codepages anymore.
>
> It seems TK windows are Unicode, but the message pump is implemented with
> the ANSI variants of GetMessaget() etc.
>
> I have laid out some details at comp.lang.tcl (
> https://groups.google.com/forum/#!topic/comp.lang.tcl/3HaLG78feUE)
>
>
>
> Of course I may be on the wrong track here, because I don’t have a perfect
> overview of all the settings and code paths in TK.
>
>
>
> Any hint or any help would be very much appreciated.
>
>
>
> We don’t want to activate the workaround with the custom IME editor again,
> because it implies hosting a single standard windows textbox in our TK GUI
> just for this purpose, which caused all kinds of problems.
>
>
>
> We have implemented numerous modifications to the TCL/TK 8.4 source code
> to adapt it for our needs so we need to stick to this version for now. (But
> a short test with the TCL console of ActiveState TCL 8.6 revealed the same
> problems there.)
>
>
>
> Kind regards,
>
> Robert Neumann.
>
>
>
> Vizrt Austria GmbH
>
>
>
> ------------------------------------------------------------------------------
>
> Want excitement?
>
> Manually upgrade your production database.
>
> When you want reliability, choose Perforce
>
> Perforce version control. Predictably reliable.
>
> http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
>
>
>
> _______________________________________________
>
> Tcl-Core mailing list
>
> Tcl...@li...
>
> https://lists.sourceforge.net/lists/listinfo/tcl-core
>
>
>
>
> ------------------------------------------------------------------------------
> Want excitement?
> Manually upgrade your production database.
> When you want reliability, choose Perforce
> Perforce version control. Predictably reliable.
>
> http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core
>
>