#27 Non-ASCII characters not shown in user interface

[Tobias Schönberg]

After trying a few alternative approaches and asking around in IRC and on stackoverflow I am pretty sure now that there is a bug with UTF-8 decoding/encoding in the pygi package for Windows, specifically libintl-8.dll. It causes non-ascii characters to not be shown in the user interface.

The warning in the terminal reads: Pango-Warning **: Invalid UTF-8 string passed to pango_layout_set_text()

The problem is independent of the environment variable PANGOCAIRO_BACKEND=win32 or =fontconfig (=fc). The font also doesn't seem to be the problem.

It should be noted that gettext translations of the same strings and printed to the terminal work perfectly. So my explanation of the problem is that it is either a problem in how libintl-8 handles these strings or how it passes them on to the user interface.

Debugging this is nearly impossible in python, because most of the functions that are involved to not have any helpful return values, or seem to fail silently (e.g. when the mo file of a locale is not found it seems to fail silently).

I put the code here (small example program): https://github.com/tobias47n9e/pygobject-locale

The installer of the test program is here: https://www.dropbox.com/s/hyvgslafa76qppo/pygibank_2.0.exe?dl=0

I uploaded some screenshots here: http://stackoverflow.com/questions/32182159/non-ascii-symbols-in-translation-of-gtk-gui-in-windows-not-working

[tumagonx]

Thanks for your help, here is updated 32bit libintl 0.19.5.1 if still doesn't work I might need to try gnu libiconv (previously win-iconv) but that would need full rebuild of pygi.

[Tobias Schönberg]

The 32bit of that file launches without errors, but the user interface stays in the original English all the time (Translation seems to work because German is printed to the terminal). But I don't want to rule out a mistake on my side. Can you maybe attach the 64-bit version too, so I can test with my normal setup?

[tumagonx]

Hmm I should did this earlier.. I have crosscheck with mypaint (packaged by me) and quodlibet (packaged by lazka) but can't reproduce your issue with German locale. Here the screenshot, both are python 2.7, quodlibet with pygi 3.10 (pangocairo-win32) and mypaint with pygi 3.14 (pangocairo-fc). Additionally All C apps such glade, gda, gedit show correct unicode. Now I doubt it's libintl/iconv issue.

BTW I advised you to define your localepath not realying on libintl's guessed path.

[Tobias Schönberg]

BTW I advised you to define your localepath not realying on libintl's guessed path.

If you have the time, could you point me to the line in the code where I do that. I am having trouble understanding where this is happening. Waht path does libintl guess?

[Tobias Schönberg]

I now also tried building Python 2.7 versions of the program, both with the old rev13 and rev21 (and both 64-bit). Couldn't notice any different behaviour. There are just too many moving parts at the moment for me to properly isolate the bug.

In addition to Python version, bitness, package-revision, I have also looked into the encoding of the mo files. Poedit allows to set the 'encoding' and the 'encoding of the sourcecode'. I tried UTF-8 and cp1252 (what python reports as the encoding) but they have the same result.

Thanks for testing with those 2 programs. I will look at their code and see if I can find a clue or my mistake.

[tumagonx]

When you say sometime intl can't find .mo files I think you pass the wrong localepath to bindtextdomain()?
FYI in mypaint the code related to localization is \mypaint.py and \lib\i18n.py

[Tobias Schönberg]

I now go around that problem by placing the mo file (with the application name) into gnome/share/locale/de/LC_MESSAGES/pygibank.mo. It finds the translations but it shows these Pango errors.

[tumagonx]

Has the problem identified? Should I close this?

[Tobias Schönberg]

I couldn't get it to work yet. The Pango warning is the most useful output I can get, so I also don't have a good way of debugging the error. One of the developers told me that I should file a bug for the missing return values. So I probably have to wait until that gets sorted out: https://bugzilla.gnome.org/show_bug.cgi?id=753991
Another thing I am going to try is to package the program with Py3.5.

[Tobias Schönberg]

Bajusz Tamás found a workaround for the bug:

https://github.com/tobias47n9e/pygobject-locale/issues/1

I uploaded the working example packaged with pynsist here:

https://www.dropbox.com/s/cwmowy8oh6l9n99/pygibank-py3.4.4-rev22-64b.exe?dl=0

One can change between English and German locale and the GUI gets translated correctly. All non-ASCII characters are shown. Commenting out the line self.translate_gui() makes the translation fail again. My conclusion is that there is something going wrong with the translation of the GUI if only set_translation_domain() is used.

[tumagonx]

Thanks for isolating the issue! I'll try it out

[tumagonx]

My setup:

OS: Win8.1 en-US
Python 3.4 32bit, pygi rev22

From Language Preference I set Region -> Format -> Germany

modify pygibank.py:
- Commenting out self.translate_gui()
- Change LIB_INTL to "libintl-8.dll" cause the file already in PATH environment

from root folder, run C:\python34\python -m pygibank

here is the output:
de_DE cp1252
German_Germany.1252
57783840
None
PyGiBank - The window title
Hellö User
Press the Button
On or off?

And the gui still show the correct ö char too, no error message regarding pango too

I don't understand the problem here, could it be not affect English Windows?
I will try build libintl will gnu libiconv + minimal gtk runtime so you can test it

[tumagonx]

Here is python 3.4 64bit's libintl with gnu libiconv
somehow I manage to create drop-in replacement libintl without have to rebuild the whole gtk

[Tobias Schönberg]

My last message did not get posted, so here is some of my information: I am also using an English Version of Windows 7, 64bit. With thev rev22 package I can set the locale to German or English and the GUI will be translated correctly (both using the function by Bajusz Tamás): https://github.com/tobias47n9e/pygobject-locale/blob/master/pygibank/pygibank.py#L59

Without that function the translation fails for German and according to Bajusz also for Hungarian. But there are no UTF-8-encoding errors anymore (Those happend with rev19 or rev20 I think).

Now I tried the libintl-8 file you posted above. It did not change the behaviour that I have with the rev22 package. The translation works with the additional function and fails without it.

Did you try setting your locale to "German (Germany)" in the Windows system control?

[tumagonx]

by "locale" I assume "format" in "region/language" tab as I described in my earlier comment?
I try install win7 on VM but only have 32bit one.

here is screenshot.
have you tried in VM?
I've test it to native XP 32bit and native Win8.1 64bit with the same procedure and unable to reproduce

[Tobias Schönberg]

I put an "ö" in the English version to catch any UTF-8 errors. Your screenshot shows that the translation failed.

Here is what I have with the function by Bajusz enabled:

https://www.dropbox.com/s/dgddm4e67xibvu3/pygibank%20translation%20working.png?dl=0

[tumagonx]

Sorry for my ignorance, I didn't download your file cause it's too big for my mobile connection (it also contain my own binaries) so I download from your github not realizing your repo missing precompiled de.mo file and jump into conclusion...

Could you give me only de.mo file cause your de.po file in github is in ANSI

[tumagonx]

^nevermind I could reproduce it now

[Tobias Schönberg]

No worries. I am certainly not very good at describing the problem.

Here is the link to the mo-file (You can unpack it in the pygibank folder)

https://www.dropbox.com/s/kve67bo7dqu7f8o/mo.zip?dl=0

And this is what the tree in the pygibank folder should look like:

.
├── __init__.py
├── __main__.py
├── mo
│   └── de
│       └── LC_MESSAGES
│           └── pygibank.mo
├── po
│   ├── de.po
│   └── pygibank.pot
├── pygibank.py
└── ui.glade

If that doesn't work it might need to go in to the gnome/share/locale/ folder again, but I think at the moment it works from within the pygibank folder.

[tumagonx]

I've observed your code and I think it's already follow what upstream bugreport pointed out https://bugzilla.gnome.org/show_bug.cgi?id=574520

Still I believe it should works just like mypaint did (it also use translatable string in glade files). Possible advise: separating init py file (for intl) and then call gui py file (those use from gi repository*). Also try to see working example code such in gramps/mypaint again, I might be missed something.