#29 Patch to enable Unicode support on ZMySQLDA

[Graeme Mathieson]

Hi,

I have a rudimentary patch which appears to enable
Unicode support, allowing you to specify an encoding
for Unicode strings passed into the database adapter.
I'm not sure if doing s.encode() on strings passed into
the string_literal() function is a really good idea
(maybe you can get away with evil things that won't be
correctly escaped now?) but it appears to work.

It also doesn't test that the underlying database is
happy with Unicode -- something I think only turned up
in 4.1 onwards?

[Graeme Mathieson]

Patch to add rudimentary Unicode support.

[Andy Dustman]

Logged In: YES
user_id=71372

You don't say what version your patch is against, but this
looks like 2.0.8. This should be unnecesary; try 2.0.9b3
instead. What you are doing seems very unlikely to work,
anyway. You need to call self.db.character_set_name() to
find out what character set is in use by the connection.

[Graeme Mathieson]

Logged In: YES
user_id=12789

Yep, my patch was against 2.0.8 and yes, it's a bit icky. :-)

I've retried my test with 2.0.9b3 and it's having the same
result -- trying to encode Unicode strings with the ascii
codec and failing miserably. Traceback:

2005-11-04 11:58:28 ERROR Zope.SiteErrorLog
http://localhost:8080/logicalware/my/lp/mail/migrate
Traceback (most recent call last):
File "ZPublisher/Publish.py", line 113, in publish
File "ZPublisher/mapply.py", line 88, in mapply
File "ZPublisher/Publish.py", line 40, in call_object
File
"/Users/mathie/zope-instances/migrate/Products/MailManager/migration.py",
line 319, in migrate
html_body_charset))
File "Shared/DC/ZRDB/DA.py", line 441, in call
File "DocumentTemplate/DT_String.py", line 476, in call
File "Shared/DC/ZRDB/sqlvar.py", line 138, in render
File
"/Users/mathie/zope-instances/migrate/Products/ZMySQLDA/DA.py",
line 131, in sql_quote__
return self._v_database_connection.string_literal(v)
File
"/Users/mathie/zope-instances/migrate/Products/ZMySQLDA/db.py",
line 344, in string_literal
def string_literal(self, s): return
self.db.string_literal(s)
UnicodeEncodeError: 'ascii' codec can't encode character
u'\xf3' in position 2995: ordinal not in range(128)

What I really want to do is make sure that the current
character set name for the connection definitely is UTF-8,
and make sure that all strings passed down for storage in
the database are stored as UTF-8 strings. (We're dealing
with email with lots of different character sets -- mostly
spam! -- and want to treat everything as unicode and the
backend to treat it as UTF-8 encoded. Am I going about
this entirely the wrong way?)

Thanks!

[Andy Dustman]

Logged In: YES
user_id=71372
Originator: NO

Fixed in branch/ZMySQLDA-2