From: Nathan E. <ned...@um...> - 2006-09-26 15:03:19
|
I'm at a loss. I hope y'all can help. I'm just trying to get a development version of SQLObject checked out and passing all tests, so I can work on getting some patches up to snuff and into the formal release cycle. Whether I check out 0.7.1, 0.7-bugfix, or trunk I get failed tests. Here is what (I think!) I executed: rm -f SQLObject svn co http://svn.colorstudy.com/SQLObject/trunk SQLObject cd SQLObject python setup.py develop cd sqlobject vi conftest.py # change connection URI's for sqlite and mysql py.test -D "sqlite" -> tests finished: 87 passed, 95 failed in 4.11 seconds py.test -D "mysql" -> tests finished: 170 passed, 12 failed in 7.89 seconds rm -rf SQLObject svn co http://svn.colorstudy.com/SQLObject/tags/0.7.1 SQLObject cd SQLObject python setup.py develop cd sqlobject vi conftest.py # change connection URI's for sqlite and mysql py.test -D "sqlite" -> tests finished: 78 passed, 80 failed in 199.85 seconds py.test -D "mysql" -> tests finished: 149 passed, 9 failed in 6.04 seconds py.test -D "mysql" -> tests finished: 147 passed, 11 failed in 4.83 seconds I expected at least 0.7.1 to complete all sqlite tests without failures. What gives? nathan -- Nathan Edwards, Ph.D. Center for Bioinformatics and Computational Biology 3119 Biomolecular Sciences Bldg. #296 University of Maryland, College Park, MD 20742 Phone: +1 301-405-9901 Email: ned...@um... WWWeb: http://www.umiacs.umd.edu/~nedwards |
From: Oleg B. <ph...@ph...> - 2006-09-26 16:06:18
|
On Tue, Sep 26, 2006 at 11:02:28AM -0400, Nathan Edwards wrote: > I'm just trying to get a development version of SQLObject checked out > and passing all tests, so I can work on getting some patches up to snuff > and into the formal release cycle. You are welcome! > Whether I check out 0.7.1, 0.7-bugfix, or trunk I get failed tests. Strange... > vi conftest.py # change connection URI's for sqlite and mysql You don't need to edit it - just pass an URI to -D: py.test test_basic.py -D sqlite:///home/sqlobject/test.db?debug=1 > I expected at least 0.7.1 to complete all sqlite tests without failures. All tests passed here on SQLite and Postgres, with Python 2.3, 2.4 and 2.5. Please investigate the reasons, look at logs... BTW, I have developed a suite of shell scripts to run a test, a number of tests or a the entire test suite at once - if you are interested I can send it to you. Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Nathan E. <ned...@um...> - 2006-09-26 16:55:28
|
At least I wasn't messing up something basic elsewhere... > Please investigate the reasons, look at logs... svn co http://svn.colorstudy.com/SQLObject/tags/0.7.1 SQLObject py.test -D "sqlite:///tmp/sqlite.data" -x Exception is thrown here: def convert_date(val): E return datetime.date(*map(int, val.split("-"))) > ValueError: invalid literal for int(): 26 12:29:12 [/.../lib/python2.4/site-packages/pysqlite2/dbapi2.py:63] Due to this test: def test_mxDateTime(): setupClass(DateTime2) _now = now() > dt2 = DateTime2(col1=_now, col2=_now, col3=Time(_now.hour, _now.minute, int(_now.second))) [/.../SQLObject/sqlobject/tests/test_datetime.py:60] SQLite is version 3.3.7. pysqlite is version 2.3.2. There is a difference (apparently) between > foreach a (tests/test_*.py) > py.test -D "sqlite:///tmp/sqlite.data" $a > end and > py.test -D "sqlite:///tmp/sqlite.data" Using the first method I get only the above failure, while using the second, I get 37 failures. > BTW, I have developed a suite of shell scripts to run a test, a number > of tests or a the entire test suite at once - if you are interested I can > send it to you. That would be helpful. I'm finding the learning curve pretty steep. nathan -- Nathan Edwards, Ph.D. Center for Bioinformatics and Computational Biology 3119 Biomolecular Sciences Bldg. #296 University of Maryland, College Park, MD 20742 Phone: +1 301-405-9901 Email: ned...@um... WWWeb: http://www.umiacs.umd.edu/~nedwards |
From: Oleg B. <ph...@ph...> - 2006-10-11 13:50:45
|
Hello! On Tue, Sep 26, 2006 at 12:54:35PM -0400, Nathan Edwards wrote: > py.test -D "sqlite:///tmp/sqlite.data" -x > > Exception is thrown here: > > def convert_date(val): > E return datetime.date(*map(int, val.split("-"))) > > ValueError: invalid literal for int(): 26 12:29:12 > > [/.../lib/python2.4/site-packages/pysqlite2/dbapi2.py:63] PySQLite2 returns date+time even for date fields. THere is a code in the test that accomodates to that behaviour: if getConnection().dbName == "sqlite": from sqlobject.sqlite.sqliteconnection import using_sqlite2 if using_sqlite2: # PySQLite2 returns full date/time for a date dateFormat = "%Y-%m-%d %H:%M:%S" > Due to this test: > > def test_mxDateTime(): > setupClass(DateTime2) > _now = now() > > dt2 = DateTime2(col1=_now, col2=_now, col3=Time(_now.hour, > _now.minute, int(_now.second))) > > [/.../SQLObject/sqlobject/tests/test_datetime.py:60] Can you run py.test -D "sqlite:///tmp/sqlite.data" -x -s >log 2>&1 and show the log? My log is here: ============================= test process starts ============================= testing-mode: inprocess executable: /usr/local/bin/python (2.4.3-final-0) using py lib: /usr/local/src/Python/py-dist/py <rev 24040> test_datetime.py[2] .. ================== tests finished: 2 passed in 0.09 seconds =================== inserting into sys.path: /usr/local/src/Python/py-dist 1/QueryOne: SELECT tbl_name FROM sqlite_master WHERE type='table' AND tbl_name = 'date_time1' 1/QueryR : SELECT tbl_name FROM sqlite_master WHERE type='table' AND tbl_name = 'date_time1' 2/QueryOne: SELECT tbl_name FROM sqlite_master WHERE type='table' AND tbl_name = 'date_time1' 2/QueryR : SELECT tbl_name FROM sqlite_master WHERE type='table' AND tbl_name = 'date_time1' 3/QueryOne: SELECT tbl_name FROM sqlite_master WHERE type='table' AND tbl_name = 'date_time1' 3/QueryR : SELECT tbl_name FROM sqlite_master WHERE type='table' AND tbl_name = 'date_time1' 4/Query : CREATE TABLE date_time1 ( id INTEGER PRIMARY KEY, col1 TIMESTAMP, col2 DATE, col3 TIME ) 4/QueryR : CREATE TABLE date_time1 ( id INTEGER PRIMARY KEY, col1 TIMESTAMP, col2 DATE, col3 TIME ) 5/QueryIns: INSERT INTO date_time1 (col2, col3, col1) VALUES ('2006-10-11', '17:44:07', '2006-10-11 17:44:07') 6/QueryOne: SELECT col1, col2, col3 FROM date_time1 WHERE id = (1) 6/QueryR : SELECT col1, col2, col3 FROM date_time1 WHERE id = (1) 7/QueryOne: SELECT tbl_name FROM sqlite_master WHERE type='table' AND tbl_name = 'date_time2' 7/QueryR : SELECT tbl_name FROM sqlite_master WHERE type='table' AND tbl_name = 'date_time2' 8/QueryOne: SELECT tbl_name FROM sqlite_master WHERE type='table' AND tbl_name = 'date_time2' 8/QueryR : SELECT tbl_name FROM sqlite_master WHERE type='table' AND tbl_name = 'date_time2' 9/QueryOne: SELECT tbl_name FROM sqlite_master WHERE type='table' AND tbl_name = 'date_time2' 9/QueryR : SELECT tbl_name FROM sqlite_master WHERE type='table' AND tbl_name = 'date_time2' 10/Query : CREATE TABLE date_time2 ( id INTEGER PRIMARY KEY, col1 TIMESTAMP, col2 DATE, col3 TIME ) 10/QueryR : CREATE TABLE date_time2 ( id INTEGER PRIMARY KEY, col1 TIMESTAMP, col2 DATE, col3 TIME ) 11/QueryIns: INSERT INTO date_time2 (col2, col3, col1) VALUES ('2006-10-11 17:44:07', '17:44:07', '2006-10-11 17:44:07') 12/QueryOne: SELECT col1, col2, col3 FROM date_time2 WHERE id = (1) 12/QueryR : SELECT col1, col2, col3 FROM date_time2 WHERE id = (1) Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Oleg B. <ph...@ph...> - 2006-10-11 19:39:07
|
Can you test the following patch: Index: sqlobject/sqlite/sqliteconnection.py =================================================================== --- sqlobject/sqlite/sqliteconnection.py (revision 2005) +++ sqlobject/sqlite/sqliteconnection.py (working copy) @@ -38,7 +38,7 @@ import warnings warnings.warn(DeprecationWarning("pysqlite2 does not support the encoding option")) opts["detect_types"] = sqlite.PARSE_DECLTYPES - for col_type in "text", "char", "varchar": + for col_type in "text", "char", "varchar", "date", "time", "datetime", "timestamp": sqlite.register_converter(col_type, stop_pysqlite2_converting_strings_to_unicode) sqlite.register_converter(col_type.upper(), stop_pysqlite2_converting_strings_to_unicode) try: Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Nathan E. <ned...@um...> - 2006-09-26 20:00:00
|
More test failures: py.test -D mysql tests/test_datetime.py def test_dateTime(): setupClass(DateTime1) _now =3D datetime.now() _today =3D date.today() > dt1 =3D DateTime1(col1=3D_now, col2=3D_today, col3=3D_now.time()= ) [/.../SQLObject/sqlobject/tests/test_datetime.py:22] ... > Invalid: expected an date/time string of the '%H:%M:%S'=20 format in the DateTimeCol 'col3', got <type 'datetime.timedelta'>=20 datetime.timedelta(0, 56256) instead [/.../SQLObject/sqlobject/col.py:947] 1/Query : CREATE TABLE date_time1 ( id INT PRIMARY KEY AUTO_INCREMENT, col1 DATETIME, col2 DATE, col3 TIME ) 1/QueryIns: INSERT INTO date_time1 (col2, col3, col1) VALUES=20 ('2006-09-26', '15:37:36', '2006-09-26 15:37:36') 1/QueryIns-> 1L 1/QueryOne: SELECT col1, col2, col3 FROM date_time1 WHERE id =3D (1) 1/QueryOne-> (datetime.datetime(2006, 9, 26, 15, 37, 36),=20 datetime.date(2006, 9, 26), datetime.timedelta(0, 56256)) tests/test_enum.py[2] .F def testNone(): > setupClass(EnumWithNone) [/.../SQLObject/sqlobject/tests/test_enum.py:28] > ProgrammingError: (1064, "You have an error in your SQL syntax;=20 check the manual that corresponds to your MySQL server version for the=20 right syntax to use near 'NULL)\n)' at line 3") [/.../python2.4/site-packages/MySQLdb/connections.py:35] - - - - - - - - - - - - - testNone: recorded stdout - - - - - - - - - -=20 - - - 1/Query : DESCRIBE enum_with_none 1/Query : DESCRIBE enum_with_none 1/Query : DESCRIBE enum_with_none 1/Query : CREATE TABLE enum_with_none ( id INT PRIMARY KEY AUTO_INCREMENT, l ENUM('a', 'bcd', 'e', NULL) ) tests/test_blob.py[1] F def test_BLOBCol(): if not supports('blobData'): return setupClass(ImageData) data =3D ''.join([chr(x) for x in range(256)]) prof =3D ImageData() > prof.image =3D data [/.../SQLObject/sqlobject/tests/test_blob.py:18] E myquery =3D unicode(query, self.encoding) > UnicodeDecodeError: 'ascii' codec can't decode byte=20 0x80 in position 167: ordinal not in range(128) [/.../SQLObject/sqlobject/mysql/mysqlconnection.py:73] - - - - - - - - - - - - test_BLOBCol: recorded stdout - - - - - - - - -=20 - - - 1/Query : DESCRIBE image_data 1/Query : DESCRIBE image_data 1/Query : DESCRIBE image_data 1/Query : CREATE TABLE image_data ( id INT PRIMARY KEY AUTO_INCREMENT, image TEXT ) 1/QueryIns: INSERT INTO image_data (image) VALUES ('emptydata') 1/QueryIns-> 1L 1/QueryOne: SELECT image FROM image_data WHERE id =3D (1) 1/QueryOne-> ('emptydata',) 1/Query : UPDATE image_data SET image =3D=20 ('\0^A^B^C^D^E^F^G\b\t\n^K^L\r^N^O^P ^Q^R^S^T^U^V^W^X^Y^ZESC^\^]^^^_=20 !"#$%&\'()*+,-./0123456789:;<=3D>?@ABCDEFGHIJKLMNO PQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~=EF=BF=BD=EF=BF=BD=EF=BF=BD= =EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF= =BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF= =BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD= =EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF= =BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF= =BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD= =EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF= =BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF= =BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD= =EF=BF=BD=EF=BF=BD =EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF= =BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF= =BD') WHERE id =3D (1) tests/test_decimal.py[2] def test_1decimal(): if not supports('decimalColumn'): return setupClass(DecimalTable) d =3D DecimalTable(col1=3D21.12) E assert d.col1 =3D=3D 21.12 > assert <DecimalTable 1L col1=3DDecimal("21.1200")>.col1 =3D=3D=20 21.120000000000001 [/.../SQLObject/sqlobject/tests/test_decimal.py:21] 1/Query : DESCRIBE decimal_table 1/Query : DESCRIBE decimal_table 1/Query : DESCRIBE decimal_table 1/Query : CREATE TABLE decimal_table ( id INT PRIMARY KEY AUTO_INCREMENT, col1 DECIMAL(6, 4) ) 1/QueryIns: INSERT INTO decimal_table (col1) VALUES (21.12000000000000= 1) 1/QueryIns-> 1L 1/QueryOne: SELECT col1 FROM decimal_table WHERE id =3D (1) 1/QueryOne-> (Decimal("21.1200"),) - - - - - - - - - - - test_1decimal: recorded stderr - - - - - - - - -=20 - - - /.../SQLObject/sqlobject/mysql/mysqlconnection.py:74: Warning: Data truncated for column 'col1' at row 1 return cursor.execute(myquery) tests/test_auto_old.py tests/test_auto.py (same error) def test_classCreate(self): if not supports('fromDatabase'): return > class OldAutoTest(SQLObject): [/.../SQLObject/sqlobject/tests/test_auto_old.py:156] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _=20 _ _ _ _ def columnsFromSchema(self, tableName, soClass): colData =3D self.queryAll("SHOW COLUMNS FROM %s" % tableName) results =3D [] for field, t, nullAllowed, key, default, extra in colData: if field =3D=3D 'id': continue colClass, kw =3D self.guessClass(t) E if self.kw['use_unicode'] and colClass is col.StringCol: > KeyError: 'use_unicode' [/.../SQLObject/sqlobject/mysql/mysqlconnection.py:153] - - - - - - - - - - - test_classCreate: recorded stdout - - - - - - - -=20 - - - 1/Query : CREATE TABLE IF NOT EXISTS old_auto_test ( auto_id INT AUTO_INCREMENT PRIMARY KEY, first_name VARCHAR(100), last_name VARCHAR(200) NOT NULL, age INT DEFAULT NULL, created DATETIME NOT NULL, happy char(1) DEFAULT 'Y' NOT NULL, long_field TEXT, wannahavefun TINYINT DEFAULT 0 NOT NULL ) 1/QueryAll: SHOW COLUMNS FROM old_auto_test 1/QueryAll-> (('auto_id', 'int(11)', 'NO', 'PRI',=20 None,'auto_increment'), ('first_name', 'varchar(100)', 'YES', '', None,=20 ''), ('last_name', 'varchar(200)', 'NO', '', '', ''), ('age', 'int(11)',=20 'YES', '', None, ''), ('created', 'datetime', 'NO', '', '', ''),=20 ('happy', 'char(1)', 'NO', '', 'Y', ''), ('long_field', 'text', 'YES',=20 '', None, ''), ('wannahavefun', 'tinyint(4)', 'NO', '', '0', '')) tests/test_unicode.py def test_create(): > setup() ... > OperationalError: (1170, "BLOB/TEXT column 'col1' used in key=20 specification without a key length") [/.../lib/python2.4/site-packages/MySQLdb/connections.py:35] - - - - - - - - - - - - test_create: recorded stdout - - - - - - - - -=20 - - - - 1/Query : DESCRIBE test_unicode 1/Query : DESCRIBE test_unicode 1/Query : DESCRIBE test_unicode 1/Query : CREATE TABLE test_unicode ( id INT PRIMARY KEY AUTO_INCREMENT, count INT NOT NULL UNIQUE, col1 TEXT NOT NULL UNIQUE, col2 TEXT ) Cheers! nathan --=20 Nathan Edwards, Ph.D. Center for Bioinformatics and Computational Biology 3119 Biomolecular Sciences Bldg. #296 University of Maryland, College Park, MD 20742 Phone: +1 301-405-9901 Email: ned...@um... WWWeb: http://www.umiacs.umd.edu/~nedwards |
From: Oleg B. <ph...@ph...> - 2006-09-27 09:59:34
|
On Tue, Sep 26, 2006 at 03:59:06PM -0400, Nathan Edwards wrote: > > Invalid: expected an date/time string of the '%H:%M:%S' > format in the DateTimeCol 'col3', got <type 'datetime.timedelta'> > datetime.timedelta(0, 56256) instead Never saw such an error. Puzzled... > > ProgrammingError: (1064, "You have an error in your SQL syntax; > check the manual that corresponds to your MySQL server version for the > right syntax to use near 'NULL)\n)' at line 3") Ha! MySQL! MySQL doesn't allow NULLs in EnumCol. There is a patch in the SF tracker: http://sourceforge.net/tracker/index.php?func=detail&aid=1496014&group_id=74338&atid=540672 and it would be helpful if you review and comment it. > [/.../SQLObject/sqlobject/tests/test_blob.py:18] > > E myquery = unicode(query, self.encoding) > > UnicodeDecodeError: 'ascii' codec can't decode byte > 0x80 in position 167: ordinal not in range(128) MySQLdb 1.2.1+? You have to specify an encodings in DB URI: mysql://host/db?debug=1&charset=utf-8&sqlobject_encoding=utf-8 for example. "charset" is for DB encoding, sqlobject_encoding is for client encoding. Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Nathan E. <ned...@um...> - 2006-09-27 13:45:42
|
>>> ProgrammingError: (1064, "You have an error in your SQL syntax; >> check the manual that corresponds to your MySQL server version for the >> right syntax to use near 'NULL)\n)' at line 3") > > Ha! MySQL! MySQL doesn't allow NULLs in EnumCol. There is a patch in the > SF tracker: > http://sourceforge.net/tracker/index.php?func=detail&aid=1496014&group_id=74338&atid=540672 > and it would be helpful if you review and comment it. Works fine. Here are some additional tests to make sure its handling of defaults etc. is correct. Index: tests/test_enum.py =================================================================== --- tests/test_enum.py (revision 1953) +++ tests/test_enum.py (working copy) @@ -19,7 +19,12 @@ Enum1._connection.module.ProgrammingError, Invalid), Enum1, l='b') - + raises( + (Enum1._connection.module.IntegrityError, + Enum1._connection.module.ProgrammingError, + Invalid), + Enum1, l=None) + class EnumWithNone(SQLObject): l = EnumCol(enumValues=['a', 'bcd', 'e', None]) @@ -27,5 +32,29 @@ def testNone(): setupClass(EnumWithNone) for l in [None, 'a', 'bcd', 'a', 'e', None]: - EnumWithNone(l=l) - + e = EnumWithNone(l=l) + assert e.l == l + +class EnumWithDefaultNone(SQLObject): + + l = EnumCol(enumValues=['a', 'bcd', 'e', None], default=None) + +def testDefaultNone(): + setupClass(EnumWithDefaultNone) + + e = EnumWithDefaultNone() + assert e.l == None + +class EnumWithDefaultOther(SQLObject): + + l = EnumCol(enumValues=['a', 'bcd', 'e', None], default='a') + +def testDefaultOther(): + setupClass(EnumWithDefaultOther) + + e = EnumWithDefaultOther() + assert e.l == 'a' + + > >> [/.../SQLObject/sqlobject/tests/test_blob.py:18] >> >> E myquery = unicode(query, self.encoding) >>> UnicodeDecodeError: 'ascii' codec can't decode byte >> 0x80 in position 167: ordinal not in range(128) > > MySQLdb 1.2.1+? You have to specify an encodings in DB URI: Yes. > mysql://host/db?debug=1&charset=utf-8&sqlobject_encoding=utf-8 > > for example. "charset" is for DB encoding, sqlobject_encoding is for > client encoding. Actually, need to use "charset=utf8&sqlobject_encoding=utf8" but still get an error: def decode(input, errors='strict'): E return codecs.utf_8_decode(input, errors, True) > UnicodeDecodeError: 'utf8' codec can't decode byte 0x80 in position 167: unexpected code byte Cheers! nathan -- Nathan Edwards, Ph.D. Center for Bioinformatics and Computational Biology 3119 Biomolecular Sciences Bldg. #296 University of Maryland, College Park, MD 20742 Phone: +1 301-405-9901 Email: ned...@um... WWWeb: http://www.umiacs.umd.edu/~nedwards |
From: Oleg B. <ph...@ph...> - 2006-09-27 13:50:43
|
On Wed, Sep 27, 2006 at 09:44:51AM -0400, Nathan Edwards wrote: > >http://sourceforge.net/tracker/index.php?func=detail&aid=1496014&group_id=74338&atid=540672 > > Works fine. Here are some additional tests to make sure its handling of > defaults etc. is correct. > > Index: tests/test_enum.py Thank you! > Actually, need to use "charset=utf8&sqlobject_encoding=utf8" That's strange. I can understand the "charset" part but not the "sqlobject_encoding". Probably, Python just allows any name - utf-8, utf8, UTF-8... > but still get an error: > > def decode(input, errors='strict'): > E return codecs.utf_8_decode(input, errors, True) > > UnicodeDecodeError: 'utf8' codec can't decode byte 0x80 in > position 167: unexpected code byte Ouch! Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Oleg B. <ph...@ph...> - 2006-10-02 14:20:36
|
Hello! And now a thing I want to discuss. On Wed, Sep 27, 2006 at 09:44:51AM -0400, Nathan Edwards wrote: > >>> ProgrammingError: (1064, "You have an error in your SQL syntax; > >>check the manual that corresponds to your MySQL server version for the > >>right syntax to use near 'NULL)\n)' at line 3") > > > > MySQL doesn't allow NULLs in EnumCol. There is a patch in the > >SF tracker: > >http://sourceforge.net/tracker/index.php?func=detail&aid=1496014&group_id=74338&atid=540672 > >and it would be helpful if you review and comment it. > > Works fine. But if the user really have asked for NULL in EnumCol - wouldn't it be better to raise an exception instead of silently remove None's from the list? It would be surprising for the user to not to see NULLs in the database without any hint from the SQLObject. So I think the way to handle this is: fix the tests to not pass None's for MySQL, but do not change the handling of EnumCol. What do you think? Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Oleg B. <ph...@ph...> - 2006-10-11 14:45:02
|
On Wed, Sep 27, 2006 at 09:44:51AM -0400, Nathan Edwards wrote: > >http://sourceforge.net/tracker/index.php?func=detail&aid=1496014&group_id=74338&atid=540672 > >and it would be helpful if you review and comment it. > > Works fine. Here are some additional tests to make sure its handling of > defaults etc. is correct. > > Index: tests/test_enum.py I combined all 3 patches, applied and committed in the revisions 2006-2008. Thank you! Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Nathan E. <ned...@um...> - 2006-09-28 16:29:40
|
I've been trying to get to the bottom of unicode issues with MySQL and the various tests that exercise this functionality - test_blob.py and test_unicode.py. I'm no expert on all of these issues, but I've gained a little understanding over the last few days. I'm using release 0.7.1, at revision 1954, MySQLdb 1.2.1+, MySQL 5.0, Python 2.4.3. The suggestion, when the test_blob.py and test_unicode.py failed, was to use the following charset=utf8&sqlobject_encoding=utf-8 in my connection URI, but I found that even this didn't work, generating essentially the same error. (This from test_blob.py, test_unicode.py similar). no extra connection URI settings: UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 167: ordinal not in range(128) with extra connection URI settings: UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 167: unexpected code byte The problem is, I believe, that python strings (<type 'str'>) are no guaranteed to be either utf-8 or ascii. The python string s = chr(128) is not valid ascii _or_ utf-8, but SQLObject (and in particular its tests) effectively assume it is. chr(128) is not valid ascii because it sets the 8th bit. chr(128) is not valid utf-8 because it should be represented with two bytes, the first of which should begin with 11. (See http://en.wikipedia.org/wiki/UTF-8). My solution (this is where my ignorance comes in) is to change the default sqlobjct_encoding to 'latin-1', rather than 'ascii' in mysql/mysqlconnection.py. latin-1 is a 256 symbol single byte string encoding, which correctly represents the range of python string instances. With this change, all of SQLObject's unicode and blob tests pass again, with no connection URI settings or special magic at the MySQL database end. On the other hand, in the process I've discovered a similar problem with sqlite's unicode handling. In test_unicode.py, a UnicodeColumn is made an alternateID, which implies unique. For MySQL, unique implies a key, which requires a length. So, col1 = UnicodeCol(alternateID=True) becomes col1 = UnicodeCol(alternateID=True,length=100) and the test passes for MySQL, handling all the Unicode stuff correctly. However, with the length argument, sqlite now fails this test. With no length argument, sqlite uses a TEXT type, while with a length argument, it uses a VARCHAR type. The TEXT type works (incorrectly!!! I believe) because it returns python strings rather than unicode strings. The VARCHAR type doesn't work (correctly!!! I believe) because it tries to coerce a python string into a unicode string (and a similar codec error is encoutered). So, I think the use of the 'ascii' encoding, where 'latin-1' is actually what is required is a bug that should be fixed. The MySQL driver is the only place this is done explicitly, but the problem the sqlite's VARCHAR makes me think that this bug is present implicitly in a variety of other places. Cheers! nathan -- Nathan Edwards, Ph.D. Center for Bioinformatics and Computational Biology 3119 Biomolecular Sciences Bldg. #296 University of Maryland, College Park, MD 20742 Phone: +1 301-405-9901 Email: ned...@um... WWWeb: http://www.umiacs.umd.edu/~nedwards |
From: Oleg B. <ph...@ph...> - 2006-09-28 17:11:18
|
On Thu, Sep 28, 2006 at 12:27:44PM -0400, Nathan Edwards wrote: > My solution (this is where my ignorance comes in) is to change the > default sqlobjct_encoding to 'latin-1' There is no such thing as default encoding... or at least there should not be. Those 'ascii' in col.py are really quick-n-dirty hacks made by MySQLdb-ignorant (me, really). I am going to replace them by connection.encoding, so you can set the "default" encoding in the DB URI. Unfortunately, I am too busy, and there are no other active developers. I'd like to see more people working on SQLObject. A person to maintain 0.7-branch, a few persons to work on the trunk and documentation... I'd like to concentrate on the paramstyle-branch, and then to work on SQLObject deep internals. Probably it should be unicode-only instead of unicode-aware. But that's a distant future... Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Ivan H. <i.h...@al...> - 2006-09-29 11:17:28
|
i cannot follow fully these many mails regarding unicode, mysql i'm using python 2.4.2 mysql 4.1.13 on linux, and MySQLdb 1.2.2 beta 1,= sqlobject is 071rc1 my system is win xp my tool is getting history records from MS Visual Sourcesafe and push them= into a mysql database. finally in a wxpython gui, in a grid you can see the history records,= modify some data, etc. i had problem with encoding as well, therefore i had to download the= MySQLdb api 1.2.2 beta 1 version. may be there is already a release this is my connection string connString =3D= "mysql://root:pwd@172.16.144.36/vss_history?use_unicode=3D1&charset=3Dutf8&= sqlobject_encoding=3Dutf-8" my small tool works perfectly, it can handle unicode characters, special= hungarian characters as well. i hope this can help you *********** REPLY SEPARATOR *********** On 28/09/2006 at 12:27 Nathan Edwards wrote: >I've been trying to get to the bottom of unicode issues with MySQL and >the various tests that exercise this functionality - test_blob.py and >test_unicode.py. > >I'm no expert on all of these issues, but I've gained a little >understanding over the last few days. > >I'm using release 0.7.1, at revision 1954, MySQLdb 1.2.1+, MySQL 5.0, >Python 2.4.3. > >The suggestion, when the test_blob.py and test_unicode.py failed, was to >use the following > > charset=3Dutf8&sqlobject_encoding=3Dutf-8 > >in my connection URI, but I found that even this didn't work, generating >essentially the same error. (This from test_blob.py, test_unicode.py >similar). > >no extra connection URI settings: > > UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position >167: ordinal not in range(128) > >with extra connection URI settings: > > UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position >167: unexpected code byte > >The problem is, I believe, that python strings (<type 'str'>) are no >guaranteed to be either utf-8 or ascii. > >The python string s =3D chr(128) is not valid ascii _or_ utf-8, but >SQLObject (and in particular its tests) effectively assume it is. > >chr(128) is not valid ascii because it sets the 8th bit. chr(128) is not >valid utf-8 because it should be represented with two bytes, the first >of which should begin with 11. (See http://en.wikipedia.org/wiki/UTF-8). > >My solution (this is where my ignorance comes in) is to change the >default sqlobjct_encoding to 'latin-1', rather than 'ascii' in >mysql/mysqlconnection.py. latin-1 is a 256 symbol single byte string >encoding, which correctly represents the range of python string instances. > >With this change, all of SQLObject's unicode and blob tests pass again, >with no connection URI settings or special magic at the MySQL database= end. > >On the other hand, in the process I've discovered a similar problem with >sqlite's unicode handling. In test_unicode.py, a UnicodeColumn is made >an alternateID, which implies unique. For MySQL, unique implies a key, >which requires a length. So, > >col1 =3D UnicodeCol(alternateID=3DTrue) > >becomes > >col1 =3D UnicodeCol(alternateID=3DTrue,length=3D100) > >and the test passes for MySQL, handling all the Unicode stuff correctly. > >However, with the length argument, sqlite now fails this test. > >With no length argument, sqlite uses a TEXT type, while with a length >argument, it uses a VARCHAR type. > >The TEXT type works (incorrectly!!! I believe) because it returns python >strings rather than unicode strings. The VARCHAR type doesn't work >(correctly!!! I believe) because it tries to coerce a python string into >a unicode string (and a similar codec error is encoutered). > >So, I think the use of the 'ascii' encoding, where 'latin-1' is actually >what is required is a bug that should be fixed. The MySQL driver is the >only place this is done explicitly, but the problem the sqlite's VARCHAR >makes me think that this bug is present implicitly in a variety of other >places. > >Cheers! > >nathan > >-- >Nathan Edwards, Ph.D. >Center for Bioinformatics and Computational Biology >3119 Biomolecular Sciences Bldg. #296 >University of Maryland, College Park, MD 20742 >Phone: +1 301-405-9901 >Email: ned...@um... >WWWeb: http://www.umiacs.umd.edu/~nedwards > >------------------------------------------------------------------------- >Take Surveys. Earn Cash. Influence the Future of IT >Join SourceForge.net's Techsay panel and you'll get the chance to share >your >opinions on IT & business topics through brief surveys -- and earn cash >http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge&CID=3DDE= VDEV >_______________________________________________ >sqlobject-discuss mailing list >sql...@li... >https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss Best regards, Ivan Horvath Chief Programmer Alcatel ISD PMS 2000 Product Team H-1116, Budapest Kondorfa u. 10. Tel.: +36 1 209 9546 Fax.: +36 1 209 9599 Mobil: +36 30 257 0235 VOIP: +49 511 676 478010 i.h...@al... Anyone who has never made a mistake has never tried anything new. /Albert Einstein/ |