[Sqlalchemy-tickets] [sqlalchemy] #2771: Full Unicode support on MySQL

[sqlalchemy <mi...@zz...>]

#2771: Full Unicode support on MySQL
-------------------------+-----------------------------------------
 Reporter:  kondi        |           Owner:  zzzeek
     Type:  defect       |          Status:  new
 Priority:  medium       |       Milestone:
Component:  cextensions  |        Severity:  no triage selected yet
 Keywords:               |  Progress State:  awaiting triage
-------------------------+-----------------------------------------
 We have a new server build with Python 2.7.5 / SqlAlchemy 0.8 / MySQL
 5.5.31.   We store a variety of Unicode strings in various tables, all
 using utf8 encoding.

 Today I've discovered that MySQL utf8 doesn't support UTF-8 encoding, but
 only the code points represented as one, two, and three byte characters,
 which omits a large subset of Unicode.  As of version 5.5.3, MySQL
 supports the entire Unicode set with a new encoding 'utf8mb4'.   However,
 we are unable to select this character set with SqlAlchemy.  I have just
 carefully traced an attempt to store a string containing the Unicode code
 points '\ud83d' and '\ude02', only to see the utf8-encoded result
 truncated to the substring preceding the first of these supplemental
 characters, producing this warning:

 /usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py:320:
 Warning: Incorrect string value: '\xF0\x9F\x98\xB7' for column 'text' at
 row 1

 Attempts to create tables with:

 __table_args__ = {'mysql_engine': 'InnoDB', 'mysql_charset': 'utf8mb4'}

 fail with:

      LookupError: unknown encoding: utf8mb4

 Has anyone had success using utf8mb4, or for that matter any approach to
 store the full Unicode character set with SqlAlchemy / MySQL?  Or are
 there plans to enhance SqlAlchemy to support full Unicode with MySQL?

--
Ticket URL: <http://www.sqlalchemy.org/trac/ticket/2771>
sqlalchemy <http://www.sqlalchemy.org/>
The Database Toolkit for Python