Menu ▾ ▴

#5833 Fix MinGW UTF-8 locale settings

Fixed
Masamichi Hosoda
Fixed_2_21_0 (565)
2020-03-15
2020-03-07
Masamichi Hosoda
No

Issue 5833/3: Doc: Add handling Unicode filenames on Windows to changes

Issue 5833/2: Add workaround for MinGW UTF-8 stat issue

When UTF-8 process locale is active,
LilyPond could not find some Unicode filenames.

stat () in msvcrt.dll returns -1 (i.e. error) with some UTF-8
filenames even if process code page is UTF-8 and the file exists.
So we get the process code page with GetACP () and, if it is UTF-8,
convert the filename to wide strings and use _wstat ().

If Universal CRT (UCRT, newer than msvcrt.dll) is used
(i.e. _UCRT is defined), no such workaround is needed.

Issue 5833/1: Add workaround for MinGW UTF-8 locale settings issue

Issue 2173 and 5828 activates UTF-8 process locale on Windows.
They allow LilyPond for Windows to use Unicode filenames.
However, when LilyPond outputs Unicode strings to tty, it garbled.

setlocale (LC_ALL, "") in msvcrt.dll sets user-defined ANSI code page
(i.e. 437 for English, 932 for Japanse, etc.)
even if process code page is UTF-8 (i.e. 65001 for all languages).
With this setting, outputting UTF-8 string becomes garbled.
So we get the process code page with GetACP () and,
if it is UTF-8, explicitly set locale's code page to UTF-8 (i.e. 65001).

If Universal CRT (UCRT, newer than msvcrt.dll) is used
(i.e. `_UCRT' is defined), no such workaround is needed.

http://codereview.appspot.com/551580045

Discussion

  • Anonymous

    Anonymous - 2020-03-07
    • Description has changed:

    Diff:
    • Needs: -->
    • Patch: new --> review
     

  • Anonymous

    Anonymous - 2020-03-07

    passes make, makecheck and a full make doc

     

  • Anonymous

    Anonymous - 2020-03-09
    • Patch: review --> countdown
     

  • Anonymous

    Anonymous - 2020-03-09

    Patch on coumntdown for March 11th.

     

  • Masamichi Hosoda

    Masamichi Hosoda - 2020-03-10
    • Description has changed:

    Diff:

    --- old
+++ new
@@ -1,10 +1,32 @@
-Fix MinGW UTF-8 locale settings
+Issue 5833/3: Doc: Add handling Unicode filenames on Windows to changes
+
+Issue 5833/2: Add workaround for MinGW UTF-8 stat issue
+
+When UTF-8 process locale is active,
+LilyPond could not find some Unicode filenames.
+
+`stat ()` in msvcrt.dll returns `-1` (i.e. error) with some UTF-8
+filenames even if process code page is UTF-8 and the file exists.
+So we get the process code page with `GetACP ()` and, if it is UTF-8,
+convert the filename to wide strings and use `_wstat ()`.
+
+If Universal CRT (UCRT, newer than msvcrt.dll) is used
+(i.e. `_UCRT` is defined), no such workaround is needed.
+
+Issue 5833/1: Add workaround for MinGW UTF-8 locale settings issue

 Issue 2173 and 5828 activates UTF-8 process locale on Windows.
 They allow LilyPond for Windows to use Unicode filenames.
 However, when LilyPond outputs Unicode strings to tty, it garbled.

-This commit fixes LilyPond's MinGW UTF-8 locale settings.
-It eliminates the garbled characters.
+`setlocale (LC_ALL, "")` in msvcrt.dll sets user-defined ANSI code page
+(i.e. 437 for English, 932 for Japanse, etc.)
+even if process code page is UTF-8 (i.e. 65001 for all languages).
+With this setting, outputting UTF-8 string becomes garbled.
+So we get the process code page with `GetACP ()` and,
+if it is UTF-8, explicitly set locale's code page to UTF-8 (i.e. 65001).
+
+If Universal CRT (UCRT, newer than msvcrt.dll) is used
+(i.e. `_UCRT' is defined), no such workaround is needed.

 http://codereview.appspot.com/551580045
    • Needs: -->
    • Type: -->
     

  • Anonymous

    Anonymous - 2020-03-10
    • Patch: new --> review
     

  • Anonymous

    Anonymous - 2020-03-10

    Passes make, make check and a full make doc

     

  • Anonymous

    Anonymous - 2020-03-13
    • Patch: review --> countdown
     

  • Anonymous

    Anonymous - 2020-03-13

    Patch on countdown for March 15th

     

  • Anonymous

    Anonymous - 2020-03-15
    • Patch: countdown --> push
     

  • Anonymous

    Anonymous - 2020-03-15

    Patch counted down - please push.

     

  • Masamichi Hosoda

    Masamichi Hosoda - 2020-03-15
    • labels: --> Fixed_2_21_0
    • status: Started --> Fixed
    • Patch: push -->
     

  • Masamichi Hosoda

    Masamichi Hosoda - 2020-03-15

    I've pushed to staging.

    commit f38a526a97ec3659dd450d9f4fbc15caa30216ec
    Issue 5833/3: Doc: Add handling Unicode filenames on Windows to changes

    commit c453b26d6eae33a537aeabd68501c00a6bf3bf5f
    Issue 5833/2: Add workaround for MinGW UTF-8 stat issue

    commit 6b34654ae7f93b9228e860c130754f83a6c35d21
    Issue 5833/1: Add workaround for MinGW UTF-8 locale settings issue