#298 Deal with LC_CTYPE=UTF-8

[Wolfgang Maier]

There is this three year old issue still open over on the Python bug tracker: http://bugs.python.org/issue18378

Basically, it is about "UTF-8" as a possible value for LC_CTYPE on OS X and the inability of Python's locale module to deal with it. Although this is essentially a Python problem I think 90 % of users who run into this issue do so by directly or indirectly using docutils, which bumps into the issue at:

/error_reporting.py:47: locale_encoding = locale.getlocale()[1] or locale.getdefaultlocale()[1]

Wouldn't it be worthwhile catching the resulting

ValueError: unknown locale: UTF-8

and simply set locale_encoding to "UTF-8" in this particular case? Would make life a bit easier for many Mac users.

[Günter Milde]

something like this?

--- error_reporting.py  (Revision 7932)
+++ error_reporting.py  (Arbeitskopie)
@@ -44,11 +44,19 @@
 except ImportError:
     locale_encoding = None
 else:

-    locale_encoding = locale.getlocale()[1] or locale.getdefaultlocale()[1]
-    # locale.getpreferredencoding([do_setlocale=True|False])
-    # has side-effects | might return a wrong guess.
-    # (cf. Update 1 in http://stackoverflow.com/questions/4082645/using-python-2-xs-locale-module-to-format-numbers-and-currency)
     try:
+        locale_encoding = locale.getlocale()[1] or locale.getdefaultlocale()[1]
+        # locale.getpreferredencoding([do_setlocale=True|False])
+        # has side-effects | might return a wrong guess.
+        # (cf. Update 1 in http://stackoverflow.com/questions/4082645/using-python-2-xs-locale-module-to-format-numbers-and-currency)
+    except ValueError, error: # OS X may set UTF-8 without language code
+        # see http://bugs.python.org/issue18378
+        # and https://sourceforge.net/p/docutils/bugs/298/
+        if "unknown locale: UTF-8" in error.message:
+            locale_encoding = "UTF-8"
+    except: # any other problems determining the locale -> use None
+        locale_encoding = None
+    try:
         codecs.lookup(locale_encoding or '') # None -> ''
     except LookupError:
         locale_encoding = None

[Wolfgang Maier]

yes, pretty much like that just that the

except ValueError, error

part would have to read

except ValueError as error

and it should check

if "unknown locale: UTF-8" in error.args

. Both for compatibility with Python3

[Günter Milde]

Thanks for the review.
Patch commited in revision 7947.

(Compatibility with Py3.x is handled by 2to3 conversion during installation,
using "as error" in the 2.x source would break 2.4 compatibility.)

Please reopen if it does not work as expected.

[Wolfgang Maier]

Sounds good to me then, thank you!