From: Oleg B. <ph...@ph...> - 2007-10-01 16:03:24
|
Hello! I'm pleased to announce the 0.9.2b1, the first beta of 0.9.2 release of SQLObject. What is SQLObject ================= SQLObject is an object-relational mapper. Your database tables are described as classes, and rows are instances of those classes. SQLObject is meant to be easy to use and quick to get started with. SQLObject supports a number of backends: MySQL, PostgreSQL, SQLite, and Firebird. It also has newly added support for Sybase, MSSQL and MaxDB (also known as SAPDB). Where is SQLObject ================== Site: http://sqlobject.org Development: http://sqlobject.org/devel/ Mailing list: https://lists.sourceforge.net/mailman/listinfo/sqlobject-discuss Archives: http://news.gmane.org/gmane.comp.python.sqlobject Download: http://cheeseshop.python.org/pypi/SQLObject/0.9.2b1 News and changes: http://sqlobject.org/News.html What's New ========== News since 0.9.1 ---------------- Bug Fixes ~~~~~~~~~ * Remove 'limit' from SelectResults after setting start/end so .clone() never sees limit again. * Fixed a bug in sqlbuilder._LikeQuoted() - call sqlrepr() on the expression to escape single quotes if the expression is a string. * Fixed a bug in Versioning - do not copy "alternateID" and "unique" attributes from the versioned table. Other Changes ~~~~~~~~~~~~~ * Removed SelectResults.__nonzero__, which was a design mistake. Raising an exception in __nonzero__() is inconsistent with other iterators (bool(iter([])) => True). * Changed the default value for 'varchar' in BLOBColumns from 'auto' to False (so that the default type for the columns in MySQL is BLOB, not TEXT). For a more complete list, please see the news: http://sqlobject.org/News.html Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Markus G. <m.g...@gm...> - 2007-10-02 07:34:12
|
Hi, the new version 0.9.2b1 added a try except block around the conversion to unicode of the query in mysqlconnection.py: if self.need_unicode and not isinstance(query, unicode): try: query = unicode(query, self.encoding) except UnicodeError: pass In which case is it necessary that the UnicodeError is silently ignored? Isn't it dangerous and misleading to swallow such an exception? Kind regards, Markus |
From: Oleg B. <ph...@ph...> - 2007-10-02 17:20:23
|
Hello! On Tue, Oct 02, 2007 at 09:34:08AM +0200, Markus Gritsch wrote: > the new version 0.9.2b1 added a try except block around the conversion > to unicode of the query in mysqlconnection.py: > > if self.need_unicode and not isinstance(query, unicode): > try: > query = unicode(query, self.encoding) > except UnicodeError: > pass > > In which case is it necessary that the UnicodeError is silently > ignored? Isn't it dangerous and misleading to swallow such an > exception? Pretty valid concern that deserves a longer answer. From time to time people report problems with MySQLdb and unicode. See this report, for example: http://sourceforge.net/mailarchive/forum.php?thread_name=54b165660707111303h45521239ncc4df928fb40a975%40mail.gmail.com&forum_name=sqlobject-discuss I never knew how to fix the problem, and was reluctant to apply different patches that were more like small hacks to fix small part of the problem. Finally David Turner dived into the problem and, using test_blob.py and test_unicode.py tests, fixed the problem; the try/except is the part of the solution. As I don't use MySQL I have to rely on the work of other people, and this time it seems David done a good job, so I blessed the inclusion of the code. Do you have a problem with the new code? Can you write a valid test case that breaks on the code? Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Markus G. <m.g...@gm...> - 2007-10-03 06:32:33
|
On 10/2/07, Oleg Broytmann <ph...@ph...> wrote: > Do you have a problem with the new code? No, so far it works fine. Markus |
From: Markus G. <m.g...@gm...> - 2007-10-04 11:50:56
|
On 10/4/07, Oleg Broytmann <ph...@ph...> wrote: > Now where is self.need_unicode used? I don't know. > We've finally returned to the > beginning - do not use unicode with MySQLdb at all? No, I don't think this is true. In MySQLdb queries are *allowed* to be unicode. >From the 'MySQLdb User Guide' (http://mysql-python.sourceforge.net/MySQLdb.html) use_unicode [...], but you can always write Unicode strings. And from the 'API documentation ' (http://mysql-python.sourceforge.net/MySQLdb-1.2.2/public/MySQLdb.connections.Connection-class.html) use_unicode [...] Unicode objects will always be encoded to the connection's character set regardless of this setting. Markus |
From: Oleg B. <ph...@ph...> - 2007-10-04 12:08:33
|
On Thu, Oct 04, 2007 at 01:50:52PM +0200, Markus Gritsch wrote: > On 10/4/07, Oleg Broytmann <ph...@ph...> wrote: > > We've finally returned to the > > beginning - do not use unicode with MySQLdb at all? > > No, I don't think this is true. In MySQLdb queries are *allowed* to be unicode. I meant - there were a period when SQLObject forces queries to be unicode for MySQLdb 1.2.1+. Is that over now? Do we allow unicode but not enforce it regardless of MySQLdb version? Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Markus G. <m.g...@gm...> - 2007-10-04 12:48:00
|
On 10/4/07, Oleg Broytmann <ph...@ph...> wrote: > On Thu, Oct 04, 2007 at 01:50:52PM +0200, Markus Gritsch wrote: > > In MySQLdb queries are *allowed* to be unicode. > > I meant - there were a period when SQLObject forces queries to be > unicode for MySQLdb 1.2.1+. Is that over now? Do we allow unicode but not > enforce it regardless of MySQLdb version? Ah, I understand. Well, I have not really an idea :( Maybe it would be a good idea to decode the string using self.encoding it was the case before removing it... David? |
From: David T. <no...@op...> - 2007-10-04 21:40:49
|
On Thu, 2007-10-04 at 14:47 +0200, Markus Gritsch wrote: > On 10/4/07, Oleg Broytmann <ph...@ph...> wrote: > > On Thu, Oct 04, 2007 at 01:50:52PM +0200, Markus Gritsch wrote: > > > In MySQLdb queries are *allowed* to be unicode. > > > > I meant - there were a period when SQLObject forces queries to be > > unicode for MySQLdb 1.2.1+. Is that over now? Do we allow unicode but not > > enforce it regardless of MySQLdb version? > > Ah, I understand. Well, I have not really an idea :( Maybe it would > be a good idea to decode the string using self.encoding it was the > case before removing it... David? I spent nearly all day working on this. I have to admit that I don't really understand all of the layers fully. Here's what I thought this morning, not because I understand what anything does, but because that's what makes the tests pass: All queries should be plain strings encoded in various arbitrary and sometimes conflicting methods. But this is totally not true. The truth is much more complicated. Imagine you have a table with one latin-1 column and one utf-8 column, and you want to update both columns. How are you going to send that to mysql? You'll might think to send it like this (assuming you want both to be character 0xf1 ): "update morx set fleem = '\xf1', baz = '\xc3\xb1';" Where fleem is the latin-1 column and baz is the utf-8 column. This doesn't actually work, because MySQL expects you to talk to it in some fixed charset -- which is set per-connection. And this leads into the bug that I found: When a column is encoded in UTF-16, everything goes horribly wrong. I was thinking this would be pretty easy to fix, but it's not, since there are so many places that encoding and decoding happen: into sqlobject, on the column, on the query (formerly) in sqlobject, on the query in mysqldb, maybe inside the mysql client library, on the mysql server. There's this additional connection charset parameter which can differ from the column charset. The connection charset is what determines how data is sent into mysql. In the test, UTF-8 happens to work because treating UTF-8 as Latin-1 (the connection charset) doesn't break anything roundtrip. UTF-16 doesn't work, because while we can insert UTF-16 into a latin-1 string, it doesn't make the whole string UTF-16. So when we try to convert back, we get latin-1 treated as UTF-16. Not good. So you might think that we should just make our connection charset always utf-8, and then have our columns convert their utf-16 to utf-8. But the way that MySQLdb sets the charset is via "SET NAMES", which also sets the collation to the default collation of that charset -- whether or not that collation works for the charset of your columns! Using SET CHARACTER SET instead fixes that, but breaks further on -- non-ascii characters (which are in latin-1) are being inserted into latin-1 columns as '?'. This is probably because when we send them across the connection, we don't encode them in any way, so they end up treated as invalid utf-8. You might ask why we don't go back to encoding the whole query -- that's because then we would be double-encoding the stuff that really is utf-8 -- and we still don't know what we're converting the query *from*. I believe I am going slightly crazy here. I will think about this some more, but possibly not immediately. In the meantime, if anyone has any thoughts for how to fix this encoding madness, they would be appreciated. |
From: David T. <no...@op...> - 2007-10-05 16:27:09
|
On Thu, 2007-10-04 at 17:40 -0400, David Turner wrote: > On Thu, 2007-10-04 at 14:47 +0200, Markus Gritsch wrote: > > On 10/4/07, Oleg Broytmann <ph...@ph...> wrote: > > > On Thu, Oct 04, 2007 at 01:50:52PM +0200, Markus Gritsch wrote: > > > > In MySQLdb queries are *allowed* to be unicode. > > > > > > I meant - there were a period when SQLObject forces queries to be > > > unicode for MySQLdb 1.2.1+. Is that over now? Do we allow unicode but not > > > enforce it regardless of MySQLdb version? > > > > Ah, I understand. Well, I have not really an idea :( Maybe it would > > be a good idea to decode the string using self.encoding it was the > > case before removing it... David? > > I spent nearly all day working on this. > > I have to admit that I don't really understand all of the layers fully. Wait, after sleeping on it (and having extremely unsettling dreams), I think I may have it: The charset for a column is the charset that the column is stored in inside mysql. It has nothing whatsoever to do with SQLObject, if I understand correctly, except that SQLObject needs to set it when the table is created. It doesn't presently do so, it seems. Right now, we act as though we need to encode data into the column format. This is wrong. We should just send everything to mysql in the connection encoding (which we should set to always be utf-8, since it can represent anything). Mysql will convert it to the column encoding. I don't have time right now to do this, but now that I know where to go, I might get to it some time. Or if someone else is bored/needs it sooner. |
From: Markus G. <m.g...@gm...> - 2007-11-27 13:02:12
Attachments:
mysqlconnection.patch
|
On 04/10/2007, Oleg Broytmann <ph...@ph...> wrote: > On Wed, Oct 03, 2007 at 06:15:21PM -0400, David Turner wrote: > > On Wed, 2007-10-03 at 23:57 +0200, Markus Gritsch wrote: > > > Ah, I see. I have no objection against removing the conversion to > > > unicode completely. > > > > Ok, I committed it to trunk. Thanks for your suggestions. > > Now where is self.need_unicode used? We've finally returned to the > beginning - do not use unicode with MySQLdb at all? How about applying the attached patch? Markus |
From: Oleg B. <ph...@ph...> - 2007-11-27 13:09:36
|
On Tue, Nov 27, 2007 at 02:02:02PM +0100, Markus Gritsch wrote: > How about applying the attached patch? --- mysqlconnection_orig.py 2007-09-27 02:18:28.000000000 +0200 +++ mysqlconnection.py 2007-11-27 13:57:30.324125000 +0100 @@ -110,10 +110,7 @@ # For MySQLdb 1.2.1 and later, we go # encoding->unicode->charset (in the mysql db) if self.need_unicode and not isinstance(query, unicode): - try: - query = unicode(query, self.encoding) - except UnicodeError: - pass + query = unicode(query, self.encoding) return cursor.execute(query) except MySQLdb.OperationalError, e: if e.args[0] in (MySQLdb.constants.CR.SERVER_GONE_ERROR, MySQLdb.constants.CR.SERVER_LOST): I remember people had been persuading me to add that try/except for years, and now you are starting the reverse process! :) I am afraid something is deeply broken in SQLObject regarding MySQL+unicode. Wonder what and how to fix that really. Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Andres F. <an...@an...> - 2007-10-03 16:05:59
|
Hi, On Tuesday 02 October 2007, Oleg Broytmann wrote in "Re: [SQLObject] SQLObj= ect=20 0.9.2b1": > Hello! > > On Tue, Oct 02, 2007 at 09:34:08AM +0200, Markus Gritsch wrote: > > the new version 0.9.2b1 added a try except block around the conversion > > to unicode of the query in mysqlconnection.py: > > > > if self.need_unicode and not isinstance(query, unicode): > > try: > > query =3D unicode(query, self.encoding) > > except UnicodeError: > > pass > > > > In which case is it necessary that the UnicodeError is silently > > ignored? Isn't it dangerous and misleading to swallow such an > > exception? > > Pretty valid concern that deserves a longer answer. > > From time to time people report problems with MySQLdb and unicode. See > this report, for example: > ... > I never knew how to fix the problem, and was reluctant to apply > different patches that were more like small hacks to fix small part of > the problem. > Finally David Turner dived into the problem and, using test_blob.py and > test_unicode.py tests, fixed the problem; the try/except is the part of t= he > solution. Perhaps it would be nice to include that description or the one of David in= a=20 comment? Those statements surely raise a warning light in the head of many= =20 people... Greetings, Andres |
From: David T. <no...@op...> - 2007-10-03 20:45:48
|
On Wed, 2007-10-03 at 11:22 +0200, Andres Freund wrote: > Hi, > > On Tuesday 02 October 2007, Oleg Broytmann wrote in "Re: [SQLObject] SQLObject > 0.9.2b1": > > Hello! > > > > On Tue, Oct 02, 2007 at 09:34:08AM +0200, Markus Gritsch wrote: > > > the new version 0.9.2b1 added a try except block around the conversion > > > to unicode of the query in mysqlconnection.py: > > > > > > if self.need_unicode and not isinstance(query, unicode): > > > try: > > > query = unicode(query, self.encoding) > > > except UnicodeError: > > > pass > > Finally David Turner dived into the problem and, using test_blob.py and > > test_unicode.py tests, fixed the problem; the try/except is the part of the > > solution. > Perhaps it would be nice to include that description or the one of David in a > comment? Those statements surely raise a warning light in the head of many > people... I was thinking that it might make more sense to just remove that whole section. I understand that whoever wrote that code thought it was necessary to convert to Unicode, but I think it is not. Can you comment on this idea? |
From: Oleg B. <ph...@ph...> - 2007-10-03 20:49:19
|
On Wed, Oct 03, 2007 at 04:46:00PM -0400, David Turner wrote: > > > > if self.need_unicode and not isinstance(query, unicode): > > > > try: > > > > query = unicode(query, self.encoding) > > > > except UnicodeError: > > > > pass > > I was thinking that it might make more sense to just remove that whole > section. I understand that whoever wrote that code thought it was > necessary to convert to Unicode, but I think it is not. Can you comment > on this idea? Yes, that is the idea - there are reports that MySQLdb 1.2.1+ requires unicode. Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Markus G. <m.g...@gm...> - 2007-11-27 13:43:11
|
On 27/11/2007, Oleg Broytmann <ph...@ph...> wrote: > On Tue, Nov 27, 2007 at 02:02:02PM +0100, Markus Gritsch wrote: > > How about applying the attached patch? > > --- mysqlconnection_orig.py 2007-09-27 02:18:28.000000000 +0200 > +++ mysqlconnection.py 2007-11-27 13:57:30.324125000 +0100 > @@ -110,10 +110,7 @@ > # For MySQLdb 1.2.1 and later, we go > # encoding->unicode->charset (in the mysql db) > if self.need_unicode and not isinstance(query, unicode): > - try: > - query = unicode(query, self.encoding) > - except UnicodeError: > - pass > + query = unicode(query, self.encoding) > return cursor.execute(query) > except MySQLdb.OperationalError, e: > if e.args[0] in (MySQLdb.constants.CR.SERVER_GONE_ERROR, MySQLdb.constants.CR.SERVER_LOST): > > I remember people had been persuading me to add that try/except for years, > and now you are starting the reverse process! :) I have been persuading you to add the "and not isinstance(query, unicode)" part of the if for month and since it was added, the MySQL backend gracefully accepts Unicode in queries ;) However, this silently swallowing of an UnicodeError was introduced from 0.9.1 -> 0.9.2 and I don't see what bug this solves. It just makes me feel uneasy. If there is really a case for which the conversion to unicode using self.encoding raises UnicodeError, then fixing it by swallowing this exception is definitely not the correct solution to the problem. Unless someone can come up with a short code example of a program which works only with the try/except being there and fails without it, I would rather like to see the try/except being removed. Markus |
From: Oleg B. <ph...@ph...> - 2007-11-27 13:49:04
|
On Tue, Nov 27, 2007 at 02:43:06PM +0100, Markus Gritsch wrote: > On 27/11/2007, Oleg Broytmann <ph...@ph...> wrote: > > if self.need_unicode and not isinstance(query, unicode): > > - try: > > - query = unicode(query, self.encoding) > > - except UnicodeError: > > - pass > > + query = unicode(query, self.encoding) [skip] > However, this silently swallowing of an UnicodeError was introduced > from 0.9.1 -> 0.9.2 and I don't see what bug this solves. I think it was added to solve problems with BLOBs, to pass the query string unmodified down to MySQLdb (I don't know how MySQLdb reacts to a non-unicode query string now). Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Markus G. <m.g...@gm...> - 2007-10-03 20:49:51
|
On 10/3/07, David Turner <no...@op...> wrote: > I was thinking that it might make more sense to just remove that whole > section. I understand that whoever wrote that code thought it was > necessary to convert to Unicode, but I think it is not. Can you comment > on this idea? There was a thread on this list containing an example which breaks without the conversion and runs fine with it. Markus |
From: David T. <no...@op...> - 2007-10-03 21:03:02
|
On Wed, 2007-10-03 at 22:49 +0200, Markus Gritsch wrote: > On 10/3/07, David Turner <no...@op...> wrote: > > I was thinking that it might make more sense to just remove that whole > > section. I understand that whoever wrote that code thought it was > > necessary to convert to Unicode, but I think it is not. Can you comment > > on this idea? > > There was a thread on this list containing an example which breaks > without the conversion and runs fine with it. Can you send me a link to this? |
From: Markus G. <m.g...@gm...> - 2007-10-03 20:58:37
|
On 10/3/07, Markus Gritsch <m.g...@gm...> wrote: > On 10/3/07, David Turner <no...@op...> wrote: > > I was thinking that it might make more sense to just remove that whole > > section. I understand that whoever wrote that code thought it was > > necessary to convert to Unicode, but I think it is not. Can you comment > > on this idea? > > There was a thread on this list containing an example which breaks > without the conversion and runs fine with it. http://www.mail-archive.com/sql...@li.../msg02607.html Markus |
From: Markus G. <m.g...@gm...> - 2007-11-27 13:57:04
|
On 27/11/2007, Oleg Broytmann <ph...@ph...> wrote: > On Tue, Nov 27, 2007 at 02:43:06PM +0100, Markus Gritsch wrote: > > On 27/11/2007, Oleg Broytmann <ph...@ph...> wrote: > > > if self.need_unicode and not isinstance(query, unicode): > > > - try: > > > - query = unicode(query, self.encoding) > > > - except UnicodeError: > > > - pass > > > + query = unicode(query, self.encoding) > [skip] > > However, this silently swallowing of an UnicodeError was introduced > > from 0.9.1 -> 0.9.2 and I don't see what bug this solves. > > I think it was added to solve problems with BLOBs, to pass the query > string unmodified down to MySQLdb (I don't know how MySQLdb reacts to > a non-unicode query string now). But if a BLOB contains just bytes which do *not* trigger the Exception, the BLOB in the query *does* get converted to unicode, and I doubt this would be the desired behavior. Markus |
From: Oleg B. <ph...@ph...> - 2007-11-27 14:02:52
|
On Tue, Nov 27, 2007 at 02:56:59PM +0100, Markus Gritsch wrote: > But if a BLOB contains just bytes which do *not* trigger the > Exception, the BLOB in the query *does* get converted to unicode, and > I doubt this would be the desired behavior. The absence of try/except in question wouldn't prevent anything in this case, right? But it helps (as I've heard) in case of a BLOB that triggers the exception. Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Markus G. <m.g...@gm...> - 2007-11-27 14:17:27
|
On 27/11/2007, Oleg Broytmann <ph...@ph...> wrote: > On Tue, Nov 27, 2007 at 02:56:59PM +0100, Markus Gritsch wrote: > > But if a BLOB contains just bytes which do *not* trigger the > > Exception, the BLOB in the query *does* get converted to unicode, and > > I doubt this would be the desired behavior. > > The absence of try/except in question wouldn't prevent anything in this > case, right? Right, but the presence of this try/except does swallow Exceptions which might be helpful in some other cases. Having this try/except here unconditionally for all cases just prevents from getting an exception when converting to unicode e.g. if one has specified a wrong encoding. This is dangerous. Instead of getting an exception, the code just continues to run, issuing a query containing the wrong characters for the DB. > But it helps (as I've heard) in case of a BLOB that triggers > the exception. But I doubt that the BLOB would be inserted *correctly* if it gets converted to unicode using the specified encoding. IMO it would be desired to insert the BLOB as it is, without trying to decode it. As already said, for example BLOBS which contain only bytes in the ASCII range would not trigger the Exception, and do get converted. The BLOB in the DB would then contain wrong bytes. Markus |
From: Oleg B. <ph...@ph...> - 2007-11-27 14:27:09
|
On Tue, Nov 27, 2007 at 03:17:23PM +0100, Markus Gritsch wrote: > Instead of getting an exception, the > code just continues to run, issuing a query containing the wrong > characters for the DB. [skip] > But I doubt that the BLOB would be inserted *correctly* if it gets > converted to unicode using the specified encoding. IMO it would be > desired to insert the BLOB as it is, without trying to decode it. This is the central part of the problem. Nobody knows for sure how different versions of MySQL and MySQLdb work according to unicode, encodings and BLOBs. Someone has to investigate (looking into code, docs, writing a number of test programs for a number of MySQL+MySQLdb variants). It could be the entire "need_unicode" in SQLObject is wrong. Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: David T. <no...@op...> - 2007-11-27 16:55:46
|
On Tue, 2007-11-27 at 15:17 +0100, Markus Gritsch wrote: > On 27/11/2007, Oleg Broytmann <ph...@ph...> wrote: > > On Tue, Nov 27, 2007 at 02:56:59PM +0100, Markus Gritsch wrote: > > > But if a BLOB contains just bytes which do *not* trigger the > > > Exception, the BLOB in the query *does* get converted to unicode, and > > > I doubt this would be the desired behavior. > > > > The absence of try/except in question wouldn't prevent anything in this > > case, right? > > Right, but the presence of this try/except does swallow Exceptions > which might be helpful in some other cases. Having this try/except > here unconditionally for all cases just prevents from getting an > exception when converting to unicode e.g. if one has specified a wrong > encoding. This is dangerous. Instead of getting an exception, the > code just continues to run, issuing a query containing the wrong > characters for the DB. But SQLObject already sends the wrong characters for the DB for MySQL by default. By default, MySQL expects data to be sent over the wire encoded in latin-1. SQLObject by default sends Unicode data encoded in UTF-8. Since UTF-8 is a subset of latin-1, this doesn't cause any exceptions. But since MySQL thinks the data is latin-1, it will return weird results for some queries: Consider a connection string like this: connectionForURI('mysql://test:test@localhost/test?use_unicode=1') class Person(SQLObject): name = UnicodeCol() sname = BLOBCol() Person.createTable() p = Person(name=u'\u20ac', sname = u'\u20ac'.encode('utf-8')) # \u20ac is the 'Euro symbol'. If you go to the mysql command line, you would expect select length(name), length(sname) from person; to return 1, 3 (3 is the length of \u20ac encoded in utf-8). In fact it returns 3, 3. So then you think you'll specify the charset: sqlhub.threadConnection = connectionForURI('mysql://test:test@localhost/test?use_unicode=1&charset=utf8&sqlobject_encoding=utf-8') Now run that same query: select length(name), length(sname) from person; 1, 1 But wait, if the length of the blob col is 1, then is the blob column being treated as utf-8? No. It's being mangled. assert Person.select()[0].sname == u'\u20ac'.encode('utf-8') #assertion fails. In fact, the result of the query is that sname contains '\x80'. Conclusion: SQLObject is doing the wrong thing on many levels for MySQL. |
From: David T. <no...@op...> - 2007-11-27 18:30:53
|
On Tue, 2007-11-27 at 18:01 +0100, Markus Gritsch wrote: > On 27/11/2007, David Turner <no...@op...> wrote: > > On Tue, 2007-11-27 at 15:17 +0100, Markus Gritsch wrote: > > > > Conclusion: SQLObject is doing the wrong thing on many levels for MySQL. > > No it is not. Using UTF-8 is an ideal default. IMO the problem in > your example is that you do not use UTF-8 in MySQL. I use UTF-8 as > default encoding in MySQL and use there parameters in the SQLObject > connection string: > ?use_unicode=1&charset=utf8&sqlobject_encoding=utf-8 > > -> Everything works flawless. > > Markus > > P.S. Thank you for your example, I will look into it in more detail. Do you mean that I need to configure MySQL itself in some way? If so, what setting do I need to change? If you mean that I need to change the table's column type, I just tried that (via mysql). Using the connection string above: select length(name), length(sname) from person; got: 3, 1 expected: 1, 3 So this is precisely backwards. |