Thread: [SQLObject] troubles with unicode
SQLObject is a Python ORM.
Brought to you by:
ianbicking,
phd
From: Petr J. <pet...@tp...> - 2012-01-29 10:06:39
|
I am lost, trying to find solution whole night long. all attempts to call the set method (see bellow) are finishing: Page handler: <bound method ProdejniMista.ulozeniProdejnihoMista of <fantomas.controllers.ProdejniMista instance at 0x2ac67a0>> Traceback (most recent call last): File "/usr/lib/python2.5/site-packages/CherryPy-2.3.0-py2.5.egg/cherrypy/_cphttptools.py", line 121, in _run self.main() File "/usr/lib/python2.5/site-packages/CherryPy-2.3.0-py2.5.egg/cherrypy/_cphttptools.py", line 264, in main body = page_handler(*virtual_path, **self.params) File "<string>", line 3, in ulozeniProdejnihoMista File "/usr/local/lib/python2.5/site-packages/TurboGears-1.0.8-py2.5.egg/turbogears/controllers.py", line 360, in expose *args, **kw) File "<string>", line 5, in run_with_transaction File "/usr/local/lib/python2.5/site-packages/TurboGears-1.0.8-py2.5.egg/turbogears/database.py", line 359, in so_rwt retval = func(*args, **kw) File "<string>", line 5, in _expose File "/usr/local/lib/python2.5/site-packages/TurboGears-1.0.8-py2.5.egg/turbogears/controllers.py", line 373, in <lambda> mapping, fragment, args, kw))) File "/usr/local/lib/python2.5/site-packages/TurboGears-1.0.8-py2.5.egg/turbogears/controllers.py", line 410, in _execute_func output = errorhandling.try_call(func, *args, **kw) File "/usr/local/lib/python2.5/site-packages/TurboGears-1.0.8-py2.5.egg/turbogears/errorhandling.py", line 77, in try_call return func(self, *args, **kw) File "/TG_web/fantomas/controllers.py", line 305, in ulozeniProdejnihoMista prodMisto.set(**slovnikArgumentu) File "/usr/lib/python2.5/site-packages/SQLObject-0.10.1-py2.5.egg/sqlobject/main.py", line 1120, in set self._connection._SO_update(self, args) File "/usr/lib/python2.5/site-packages/SQLObject-0.10.1-py2.5.egg/sqlobject/dbconnection.py", line 509, in _SO_update for dbName, value in values]), UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 33: ordinal not in range(128) I have tried all possible combination to decode/encode the field values and I am really sure all filelds are unicode fields when i am trying call the set method. automat.set(**slovnikArgumentu) automat.set(telKontaktniOsoby = slovnikArgumentu[telKontaktniOsoby].encode("utf-8"), najemZaRokBezDph = slovnikArgumentu[najemZaRokBezDph].encode("utf-8"), kontaktniOsobaPrijmeni = slovnikArgumentu[kontaktniOsobaPrijmeni].encode("utf-8"), nazevKontaktMistaPoruchy = slovnikArgumentu[nazevKontaktMistaPoruchy].encode("utf-8"), psc = slovnikArgumentu[psc].encode("utf-8"), mesto = slovnikArgumentu[mesto].encode("utf-8"), adresa2 = slovnikArgumentu[adresa2].encode("utf-8"), adresa1 = slovnikArgumentu[adresa1].encode("utf-8"), zemDelka = slovnikArgumentu[zemDelka].encode("utf-8"), zemSirka = slovnikArgumentu[zemSirka].encode("utf-8"), elektrinaZaRokBezDph = slovnikArgumentu[elektrinaZaRokBezDph].encode("utf-8"), cisloSignysPartnera = slovnikArgumentu[cisloSignysPartnera].encode("utf-8"), pmid = slovnikArgumentu[pmid].encode("utf-8"), telReseniPoruchy = slovnikArgumentu[telReseniPoruchy].encode("utf-8"), nazevProdejnihoMista = slovnikArgumentu[nazevProdejnihoMista].encode("utf-8"), dnuRokOtevreno = slovnikArgumentu[dnuRokOtevreno].encode("utf-8"), kontaktniOsobaJmeno = slovnikArgumentu[kontaktniOsobaJmeno].encode("utf-8")) My Environment: Ubuntu Hardy Heron 64bit minimal server install vs2521:/TG_web# yolk -l Cheetah - 2.4.4 - active CherryPy - 2.3.0 - active DecoratorTools - 1.8 - active Extremes - 1.1.1 - active FormEncode - 1.2.4 - active Markdown - 2.1.1 - active PIL - 1.1.6 - active Paste - 1.7.5.1 - active PasteDeploy - 1.5.0 - active PasteScript - 1.7.5 - active PyProtocols - 1.0a0dev-r2302 - active Python - 2.5.2 - active development (/usr/lib/python2.5/lib-dynload) RuleDispatch - 0.5a1.dev-r2506 - active SQLObject - 0.10.1 - active TurboCheetah - 1.0 - active TurboGears - 1.0.10 - non-active TurboGears - 1.0.8 - active development (/usr/local/lib/python2.5/site-packages/TurboGears-1.0.8-py2.5.egg) TurboJson - 1.1.4 - active TurboKid - 1.0.5 - active configobj - 4.7.2 - active egenix-mx-base - 3.2.2 - active kid - 0.9.6 - active kinterbasdb - 3.3.0 - active pip - 1.0.2 - active pygooglechart - 0.3.0 - active setuptools - 0.6c11 - non-active setuptools - 0.6c12dev-r88846 - active setuptools - 0.6c8 - non-active simplejson - 2.3.2 - active wsgiref - 0.1.2 - active development (/usr/lib/python2.5) yolk - 0.4.1 - active Thanks for your hints Petr |
From: Oleg B. <ph...@ph...> - 2012-01-29 19:29:15
|
Hello! On Sun, Jan 29, 2012 at 11:06:31AM +0100, Petr Jake?? wrote: > File "/usr/lib/python2.5/site-packages/SQLObject-0.10.1-py2.5.egg/sqlobject/main.py", > line 1120, in set > self._connection._SO_update(self, args) > File "/usr/lib/python2.5/site-packages/SQLObject-0.10.1-py2.5.egg/sqlobject/dbconnection.py", > line 509, in _SO_update > for dbName, value in values]), > UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position > 33: ordinal not in range(128) The error means there is at least one 8-bit str mixed with unicode. If you are sure all values are str instances - what about dbNames? _SO_update is called from set with dbNames from sqlmeta.columns: args = [(self.sqlmeta.columns[name].dbName, value) for name, value in toUpdate.items()] self._connection._SO_update(self, args) Make sure all dbNames are also str, not unicode. Oleg. -- Oleg Broytman http://phdru.name/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Petr J. <pet...@tp...> - 2012-01-29 20:50:08
|
> The error means there is at least one 8-bit str mixed with unicode. > If you are sure all values are str instances - what about dbNames? > _SO_update is called from set with dbNames from sqlmeta.columns: > > args = [(self.sqlmeta.columns[name].dbName, value) > for name, value in toUpdate.items()] > self._connection._SO_update(self, args) > > Make sure all dbNames are also str, not unicode. > It looks the type of some fields is changed somewhere during the code execution. When the set method is called, types of all fields are OK. When the SQLObject calls SO_update start, some of the fields and values types are changed (see the printout below). Any hints? Petr ============= set method start =============== <type 'str'> telKontaktniOsoby <type 'unicode'> +420721870631 <type 'str'> kontaktniOsobaPrijmeni <type 'unicode'> Kračmar <type 'str'> nazevKontaktMistaPoruchy <type 'unicode'> Vrátnice <type 'str'> psc <type 'unicode'> 46117 <type 'str'> mesto <type 'unicode'> Liberec <type 'str'> geocodeAccuracy <type 'str'> Z <type 'str'> adresa2 <type 'unicode'> <type 'str'> adresa1 <type 'unicode'> 17.listopadu 587/4 <type 'str'> zemDelka <type 'unicode'> 15.085940 <type 'str'> zemSirka <type 'unicode'> 50.770829 <type 'str'> elektrinaZaRokBezDph <type 'unicode'> 2052 <type 'str'> kontaktniOsobaJmeno <type 'unicode'> Zdeněk <type 'str'> cisloSignysPartnera <type 'unicode'> 8 <type 'str'> telReseniPoruchy <type 'unicode'> +420485355214 <type 'str'> nazevProdejnihoMista <type 'unicode'> Kolej LBC B <type 'str'> dnuRokOtevreno <type 'unicode'> 340 <type 'str'> najemZaRokBezDph <type 'unicode'> 5040 ============ _SO_update start ===================== <type 'str'> tel_kontaktni_osoby <type 'str'> +420721870631 <type 'str'> kontaktni_osoba_prijmeni <type 'str'> Kračmar <type 'unicode'> nazev_kontakt_mista_poruchy <type 'unicode'> Vrátnice <type 'str'> psc <type 'str'> 46117 <type 'str'> mesto <type 'str'> Liberec <type 'unicode'> geocode_accuracy <type 'str'> Z <type 'str'> adresa2 <type 'str'> <type 'str'> adresa1 <type 'str'> 17.listopadu 587/4 <type 'str'> zem_delka <type 'str'> 15.085940 <type 'str'> zem_sirka <type 'str'> 50.770829 <type 'unicode'> elektrina_za_rok_bez_dph <type 'unicode'> 2052 <type 'str'> kontaktni_osoba_jmeno <type 'str'> Zdeněk <type 'unicode'> cislo_signys_partnera <type 'unicode'> 8 <type 'str'> tel_reseni_poruchy <type 'str'> +420485355214 <type 'str'> nazev_prodejniho_mista <type 'str'> Kolej LBC B <type 'unicode'> dnu_rok_otevreno <type 'unicode'> 340 <type 'unicode'> najem_za_rok_bez_dph <type 'unicode'> 5040 |
From: Oleg B. <ph...@ph...> - 2012-01-29 21:05:12
|
On Sun, Jan 29, 2012 at 09:50:01PM +0100, Petr Jake?? wrote: > ============= set method start =============== > <type 'str'> telKontaktniOsoby <type 'unicode'> +420721870631 > <type 'str'> kontaktniOsobaPrijmeni <type 'unicode'> Kra??mar > <type 'str'> nazevKontaktMistaPoruchy <type 'unicode'> Vr??tnice [snip] > ============ _SO_update start ===================== > <type 'str'> tel_kontaktni_osoby <type 'str'> +420721870631 > <type 'str'> kontaktni_osoba_prijmeni <type 'str'> Kra??mar > <type 'unicode'> nazev_kontakt_mista_poruchy <type 'unicode'> Vr??tnice [snip] .set() converts fields to backend format using validators attached to the fields. UnicodeColumns are encoded to strings, but StringColumns are not, so when you've assigned a unicode object to a StringCol ._SO_update() received that unicode object which is wrong - ._SO_update() has to receive str. Do not assign unicode to StringCol. Oleg. -- Oleg Broytman http://phdru.name/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Petr J. <pet...@tp...> - 2012-02-23 18:50:40
|
After some investigation of the method columnsFromSchema it looks like Firebird (FB 2.5 SuperServer actualy) returns the information about the field type in following form: > u'SHORT > ' > so it is necessary to add > t = t.strip() > in the for loop (tested) > > BTW I have found with some luck the FLOAT type is not detected properly. u'FLOAT' is recognized as <class 'sqlobject.col.Col'> How to solve this? Petr |
From: Oleg B. <ph...@ph...> - 2012-02-23 20:07:22
|
On Thu, Feb 23, 2012 at 07:50:33PM +0100, Petr Jake?? wrote: > BTW I have found with some luck the FLOAT type is not detected properly. > u'FLOAT' is recognized as <class 'sqlobject.col.Col'> > > How to solve this? By extending method guessClass. Oleg. -- Oleg Broytman http://phdru.name/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Petr J. <pet...@tp...> - 2012-02-24 08:51:52
|
> By extending method guessClass. > > Oleg, I have generated the test table<https://docs.google.com/open?id=0B8Qr2m4bDZW0UjJsX0lJVnZScGE5Q2RCbFp4Z1VTQQ>for the Firebird 2.5 database. Than using the SQL select I have found on the web, I have queried fields metadata. Details here<https://docs.google.com/open?id=0B8Qr2m4bDZW0cXJYYXUyUFlTZDZtelZzWThQam5aZw> Can it be useful somehow for extending guessClass method? Petr |
From: Petr J. <pet...@tp...> - 2012-02-24 09:33:46
|
After some investigation of the method columnsFromSchema it looks like Firebird (FB 2.5 SuperServer actualy) returns the information about the field type in following form: > u'SHORT > ' > so it is necessary to add > t = t.strip() > in the for loop (tested) > > Anyway, I guess it cant hurt anything. > > My guess was wrong. It ruined a lot of things in my web app (TurboGears actually). I see on many of my pages something like bellow included Traceback. According to the Ivan Crstic, <http://farmdev.com/talks/unicode/> the schema for working with international characters = unicode has to be: 1. Decode early (by reading ASCII from the database and decode it using charset value default to the database or to the field) >>> def to_unicode_or_bust(... obj, encoding='utf-8'):... if isinstance(obj, basestring):... if not isinstance(obj, unicode):... obj = unicode(obj, encoding)... return obj 2. Unicode everywhere 3. Encode late But I am l lost again with all this UnicodeEncodeError troubles. Any idea how to solve it? What I am doing wrong? Page handler: <bound method ProdejniMista.stavyZasobniku of <fantomas.controllers.ProdejniMista instance at 0x3404e18>> Traceback (most recent call last): File "/usr/lib/python2.5/site-packages/CherryPy-2.3.0-py2.5.egg/cherrypy/_cphttptools.py", line 121, in _run self.main() File "/usr/lib/python2.5/site-packages/CherryPy-2.3.0-py2.5.egg/cherrypy/_cphttptools.py", line 264, in main body = page_handler(*virtual_path, **self.params) File "<string>", line 3, in stavyZasobniku File "/usr/local/lib/python2.5/site-packages/TurboGears-1.0.8-py2.5.egg/turbogears/controllers.py", line 360, in expose *args, **kw) File "<string>", line 5, in run_with_transaction File "/usr/local/lib/python2.5/site-packages/TurboGears-1.0.8-py2.5.egg/turbogears/database.py", line 359, in so_rwt retval = func(*args, **kw) File "<string>", line 5, in _expose File "/usr/local/lib/python2.5/site-packages/TurboGears-1.0.8-py2.5.egg/turbogears/controllers.py", line 373, in <lambda> mapping, fragment, args, kw))) File "/usr/local/lib/python2.5/site-packages/TurboGears-1.0.8-py2.5.egg/turbogears/controllers.py", line 423, in _execute_func return _process_output(output, template, format, content_type, mapping, fragment) File "/usr/local/lib/python2.5/site-packages/TurboGears-1.0.8-py2.5.egg/turbogears/controllers.py", line 88, in _process_output fragment=fragment) File "/usr/local/lib/python2.5/site-packages/TurboGears-1.0.8-py2.5.egg/turbogears/view/base.py", line 159, in render return engine.render(**kw) File "/usr/lib/python2.5/site-packages/TurboKid-1.0.5-py2.5.egg/turbokid/kidsupport.py", line 220, in render output=output, format=format) File "/usr/lib/python2.5/site-packages/kid-0.9.6-py2.5.egg/kid/__init__.py", line 301, in serialize raise_template_error(module=self.__module__) File "/usr/lib/python2.5/site-packages/kid-0.9.6-py2.5.egg/kid/__init__.py", line 299, in serialize return serializer.serialize(self, encoding, fragment, format) File "/usr/lib/python2.5/site-packages/kid-0.9.6-py2.5.egg/kid/serialization.py", line 107, in serialize text = ''.join(self.generate(stream, encoding, fragment, format)) File "/usr/lib/python2.5/site-packages/kid-0.9.6-py2.5.egg/kid/serialization.py", line 629, in generate for ev, item in self.apply_filters(stream, format): File "/usr/lib/python2.5/site-packages/kid-0.9.6-py2.5.egg/kid/serialization.py", line 165, in format_stream for ev, item in stream: File "/usr/lib/python2.5/site-packages/kid-0.9.6-py2.5.egg/kid/parser.py", line 221, in _coalesce for ev, item in stream: File "/usr/lib/python2.5/site-packages/kid-0.9.6-py2.5.egg/kid/serialization.py", line 477, in inject_meta_tags for ev, item in stream: File "/usr/lib/python2.5/site-packages/kid-0.9.6-py2.5.egg/kid/parser.py", line 179, in _track for p in stream: File "/usr/lib/python2.5/site-packages/kid-0.9.6-py2.5.egg/kid/filter.py", line 32, in apply_matches item = stream.expand() File "/usr/lib/python2.5/site-packages/kid-0.9.6-py2.5.egg/kid/parser.py", line 108, in expand for ev, item in self._iter: File "/usr/lib/python2.5/site-packages/kid-0.9.6-py2.5.egg/kid/parser.py", line 179, in _track for p in stream: File "/usr/lib/python2.5/site-packages/kid-0.9.6-py2.5.egg/kid/parser.py", line 221, in _coalesce for ev, item in stream: File "/TG_web/fantomas/templates/zasobniky.py", line 207, in _pull File "<string>", line 1, in <lambda> File "/usr/lib/python2.5/site-packages/SQLObject-0.10.1-py2.5.egg/sqlobject/main.py", line 1154, in _SO_foreignKey return joinClass.get(id, connection=self._connection) File "/usr/lib/python2.5/site-packages/SQLObject-0.10.1-py2.5.egg/sqlobject/main.py", line 882, in get val._init(id, connection, selectResults) File "/usr/lib/python2.5/site-packages/SQLObject-0.10.1-py2.5.egg/sqlobject/main.py", line 924, in _init self._SO_selectInit(selectResults) File "/usr/lib/python2.5/site-packages/SQLObject-0.10.1-py2.5.egg/sqlobject/main.py", line 1130, in _SO_selectInit colValue = col.to_python(colValue, self._SO_validatorState) File "/usr/lib/python2.5/site-packages/SQLObject-0.10.1-py2.5.egg/sqlobject/col.py", line 513, in to_python return value.encode(dbEncoding) UnicodeEncodeError: 'ascii' codec can't encode character u'\u010d' in position 8: ordinal not in range(128) Error location in template file '/TG_web/fantomas/templates/zasobniky.kid' between line 26, column 8 and line 27, column 8: <td align="left" py:content="zasobnik.ciselnik.nazev"/> |
From: Oleg B. <ph...@ph...> - 2012-02-25 15:59:53
|
On Fri, Feb 24, 2012 at 10:33:34AM +0100, Petr Jake?? wrote: > File "/usr/lib/python2.5/site-packages/SQLObject-0.10.1-py2.5.egg/sqlobject/main.py", > line 1130, in _SO_selectInit > colValue = col.to_python(colValue, self._SO_validatorState) > File "/usr/lib/python2.5/site-packages/SQLObject-0.10.1-py2.5.egg/sqlobject/col.py", > line 513, in to_python > return value.encode(dbEncoding) > UnicodeEncodeError: 'ascii' codec can't encode character u'\u010d' in > position 8: ordinal not in range(128) It seems your DB API adapter (kinterbasedb) returned unicode values. StringCol tried to encode them to strings in dbEncoding (ascii) and failed. Perhaps you have to set a proper dbEncoding. SQLObject 1.1 can set one encoding per table or per connection. Oleg. -- Oleg Broytman http://phdru.name/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Petr J. <pet...@tp...> - 2012-01-29 21:29:22
|
> > > .set() converts fields to backend format using validators attached to > the fields. UnicodeColumns are encoded to strings, but StringColumns are > not, so when you've assigned a unicode object to a StringCol > ._SO_update() received that unicode object which is wrong - > ._SO_update() has to receive str. Do not assign unicode to StringCol. > > Hmm... I am not getting it. I am on the Firebird. The database Default character set = UTF8 ALL text fields in in the table are varchar. Why the SQLObject things some columns are UnicodeColumns and some columns are StringColumns? Bingo... it looks like "Mea Culpa": Because of my laziness I am using fromDatabase = True. But now, when I have inspected the table definitions, I have found I am (accidentally) mixing two things together in one of the class/tables definition (see the code bellow). I will investigate this. Thanks a lot. Petr class ProdejniMista(SQLObject): class sqlmeta: fromDatabase = True # -> nacte vse z databaze bez potreby definice # columnList = True provozovatelMista = ForeignKey('ProvozovatelMista') automaty = RelatedJoin('Automaty') produkty = RelatedJoin('Produkty') nazevProdejnihoMista = UnicodeCol(dbEncoding = "utf-8") kontaktniOsobaJmeno = UnicodeCol(dbEncoding = "utf-8") kontaktniOsobaPrijmeni = UnicodeCol(dbEncoding = "utf-8") mesto = UnicodeCol(dbEncoding = "utf-8") adresa1 = UnicodeCol(dbEncoding = "utf-8") adresa2 = UnicodeCol(dbEncoding = "utf-8") psc = UnicodeCol(dbEncoding = "utf-8") zemDelka = UnicodeCol(dbEncoding = "utf-8") zemSirka = UnicodeCol(dbEncoding = "utf-8") telReseniPoruchy = UnicodeCol(dbEncoding = "utf-8") telKontaktniOsoby = UnicodeCol(dbEncoding = "utf-8") telKontaktniOsoby = UnicodeCol(dbEncoding = "utf-8") automaty = RelatedJoin('Automaty') |
From: Petr J. <pet...@tp...> - 2012-01-29 22:17:16
|
I thing I am few steps farther but new problem arised: File "/TG_web/fantomas/controllers.py", line 305, in ulozeniProdejnihoMista prodMisto.set(**slovnikArgumentu) File "/usr/lib/python2.5/site-packages/SQLObject-0.10.1-py2.5.egg/sqlobject/main.py", line 1123, in set self._connection._SO_update(self, args) File "/usr/lib/python2.5/site-packages/SQLObject-0.10.1-py2.5.egg/sqlobject/dbconnection.py", line 512, in _SO_update self.query(myQuery) File "/usr/lib/python2.5/site-packages/SQLObject-0.10.1-py2.5.egg/sqlobject/dbconnection.py", line 683, in query return self._dbConnection._query(self._connection, s) File "/usr/lib/python2.5/site-packages/SQLObject-0.10.1-py2.5.egg/sqlobject/dbconnection.py", line 334, in _query self._executeRetry(conn, conn.cursor(), s) File "/usr/lib/python2.5/site-packages/SQLObject-0.10.1-py2.5.egg/sqlobject/dbconnection.py", line 329, in _executeRetry return cursor.execute(query) UnicodeEncodeError: 'ascii' codec can't encode character u'\xed' in position 101: ordinal not in range(128) ============= set method start =============== <type 'str'> telKontaktniOsoby <type 'unicode'> +420734803579 <type 'str'> kontaktniOsobaPrijmeni <type 'unicode'> Zburník <type 'str'> nazevKontaktMistaPoruchy <type 'unicode'> recepce <type 'str'> psc <type 'unicode'> 28835 <type 'str'> mesto <type 'unicode'> Nymburk <type 'str'> geocodeAccuracy <type 'str'> Z <type 'str'> adresa2 <type 'unicode'> <type 'str'> adresa1 <type 'unicode'> Sportovní 1801 <type 'str'> zemDelka <type 'unicode'> 15.058984 <type 'str'> zemSirka <type 'unicode'> 50.178441 <type 'str'> elektrinaZaRokBezDph <type 'unicode'> 2356 <type 'str'> kontaktniOsobaJmeno <type 'unicode'> Tomáš <type 'str'> cisloSignysPartnera <type 'unicode'> 34 <type 'str'> telReseniPoruchy <type 'unicode'> +420325517801 <type 'str'> nazevProdejnihoMista <type 'unicode'> SC Nymburk <type 'str'> dnuRokOtevreno <type 'unicode'> 360 <type 'str'> najemZaRokBezDph <type 'unicode'> 10400 ============ _SO_update start ===================== <type 'unicode'> tel_kontaktni_osoby <type 'unicode'> +420734803579 <type 'unicode'> kontaktni_osoba_prijmeni <type 'unicode'> Zburník <type 'unicode'> nazev_kontakt_mista_poruchy <type 'unicode'> recepce <type 'unicode'> psc <type 'unicode'> 28835 <type 'unicode'> mesto <type 'unicode'> Nymburk <type 'unicode'> geocode_accuracy <type 'str'> Z <type 'unicode'> adresa2 <type 'unicode'> <type 'unicode'> adresa1 <type 'unicode'> Sportovní 1801 <type 'unicode'> zem_delka <type 'unicode'> 15.058984 <type 'unicode'> zem_sirka <type 'unicode'> 50.178441 <type 'unicode'> elektrina_za_rok_bez_dph <type 'unicode'> 2356 <type 'unicode'> kontaktni_osoba_jmeno <type 'unicode'> Tomáš <type 'unicode'> cislo_signys_partnera <type 'unicode'> 34 <type 'unicode'> tel_reseni_poruchy <type 'unicode'> +420325517801 <type 'unicode'> nazev_prodejniho_mista <type 'unicode'> SC Nymburk <type 'unicode'> dnu_rok_otevreno <type 'unicode'> 360 <type 'unicode'> najem_za_rok_bez_dph <type 'unicode'> 10400 print myQuery: UPDATE prodejni_mista SET tel_kontaktni_osoby = ('+420734803579'), kontaktni_osoba_prijmeni = ('Zburník'), nazev_kontakt_mista_poruchy = ('recepce'), psc = ('28835'), mesto = ('Nymburk'), geocode_accuracy = ('Z'), adresa2 = (''), adresa1 = ('Sportovní 1801'), zem_delka = ('15.058984'), zem_sirka = ('50.178441'), elektrina_za_rok_bez_dph = ('2356'), kontaktni_osoba_jmeno = ('Tomáš'), cislo_signys_partnera = ('34'), tel_reseni_poruchy = ('+420325517801'), nazev_prodejniho_mista = ('SC Nymburk'), dnu_rok_otevreno = ('360'), najem_za_rok_bez_dph = ('10400') WHERE id = (19) print repr(myQuerry) u"UPDATE prodejni_mista SET tel_kontaktni_osoby = ('+420734803579'), kontaktni_osoba_prijmeni = ('Zburn\xedk'), nazev_kontakt_mista_poruchy = ('recepce'), psc = ('28835'), mesto = ('Nymburk'), geocode_accuracy = ('Z'), adresa2 = (''), adresa1 = ('Sportovn\xed 1801'), zem_delka = ('15.058984'), zem_sirka = ('50.178441'), elektrina_za_rok_bez_dph = ('2356'), kontaktni_osoba_jmeno = ('Tom\xe1\u0161'), cislo_signys_partnera = ('34'), tel_reseni_poruchy = ('+420325517801'), nazev_prodejniho_mista = ('SC Nymburk'), dnu_rok_otevreno = ('360'), najem_za_rok_bez_dph = ('10400') WHERE id = (19)" Can you send me your suggestions please? Thanks Petr |
From: Petr J. <pet...@tp...> - 2012-02-25 18:10:58
|
> It seems your DB API adapter (kinterbasedb) returned unicode values. > StringCol tried to encode them to strings in dbEncoding (ascii) and > failed. Perhaps you have to set a proper dbEncoding. SQLObject 1.1 can > set one encoding per table or per connection. > > Hmmm... the unicode type string comes from the TurboGears web form actually. Now I am encoding it to the utf-8 str type when it comes from the web form but the Error occurs as before File "/usr/lib/python2.5/site-packages/SQLObject-0.10.1-py2.5.egg/sqlobject/firebird/firebirdconnection.py", line 105, in _queryInsertID q = self._insertSQL(table, names, values) File "/usr/lib/python2.5/site-packages/SQLObject-0.10.1-py2.5.egg/sqlobject/dbconnection.py", line 387, in _insertSQL ', '.join([self.sqlrepr(v) for v in values]))) UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 16: ordinal not in range(128) I made the change in the sqlobject.dburi from sqlobject.dburi="firebird://user:password@localhost :3050/pathToTheDatabase/automaty.fdb" to sqlobject.dburi="firebird://user:password@localhost :3050/pathToTheDatabase/automaty.fdb?charset=utf8" but nothing changed. I made the change in the table column definition. Because I am using "fromDatabase = True" for all tables, I added the manual definition of the column in the table definition: poznamka = UnicodeCol(dbEncoding = "utf8") but nothing changed. Do you have any idea how to force SQLObject to use the utf-8 instead of "ascii" for decoding? Petr |
From: Petr J. <pet...@tp...> - 2012-02-25 19:38:32
|
> > > Do you have any idea how to force SQLObject to use the utf-8 instead of > "ascii" for decoding? > I think I have found something. I have an other table, where inserting encoded string works. The only difference between these two tables is the field charset definition. Inserting encoded string to the table, where field charset is set to UNICODE_FSS everything works OK Inserting encoded string to the table, where field charset is set to UTF-8 throws an Exception. Any hints? Petr |
From: Oleg B. <ph...@ph...> - 2012-02-25 20:55:56
|
On Sat, Feb 25, 2012 at 07:10:51PM +0100, Petr Jake?? wrote: > sqlobject.dburi="firebird://user:password@localhost > :3050/pathToTheDatabase/automaty.fdb?charset=utf8" FirebirdConnection should rename or alias ``charset`` to ``dbEncoding``. Adding it to my TODO... Oleg. -- Oleg Broytman http://phdru.name/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Oleg B. <ph...@ph...> - 2012-01-29 22:36:17
|
On Sun, Jan 29, 2012 at 11:17:09PM +0100, Petr Jake?? wrote: > File > "/usr/lib/python2.5/site-packages/SQLObject-0.10.1-py2.5.egg/sqlobject/dbconnection.py", > line 329, in _executeRetry > return cursor.execute(query) > UnicodeEncodeError: 'ascii' codec can't encode character u'\xed' in > position 101: ordinal not in range(128) It seems Firebird DB API driver can't process unicode. > ============= set method start =============== > <type 'str'> telKontaktniOsoby <type 'unicode'> +420734803579 > <type 'str'> kontaktniOsobaPrijmeni <type 'unicode'> Zburn??k > <type 'str'> nazevKontaktMistaPoruchy <type 'unicode'> recepce > > ============ _SO_update start ===================== > <type 'unicode'> tel_kontaktni_osoby <type 'unicode'> +420734803579 > <type 'unicode'> kontaktni_osoba_prijmeni <type 'unicode'> Zburn??k > <type 'unicode'> nazev_kontakt_mista_poruchy <type 'unicode'> recepce You made all columns StringCol where I've advised you to make them all UnicodeCol. Oleg. -- Oleg Broytman http://phdru.name/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Petr J. <pet...@tp...> - 2012-01-30 00:20:24
|
> > ============= set method start =============== > > <type 'str'> telKontaktniOsoby <type 'unicode'> +420734803579 > > <type 'str'> kontaktniOsobaPrijmeni <type 'unicode'> Zburn??k > > <type 'str'> nazevKontaktMistaPoruchy <type 'unicode'> recepce > > > > ============ _SO_update start ===================== > > <type 'unicode'> tel_kontaktni_osoby <type 'unicode'> +420734803579 > > <type 'unicode'> kontaktni_osoba_prijmeni <type 'unicode'> Zburn??k > > <type 'unicode'> nazev_kontakt_mista_poruchy <type 'unicode'> recepce > > You made all columns StringCol where I've advised you to make them > all UnicodeCol. > > Hmmm.... I am completely lost :( I am testing the unicode type and replacing it with the utf-8 encoded string before I am calling the .set() method: for key, value in argumentsDict.items(): print type(key), key, type(value), value if isinstance(value, unicode): argumentsDict[key] = value.encode("utf-8") It is throwing UnicodeDecodeError anyway.... Grrrr ============= set method start =============== <type 'str'> telKontaktniOsoby value: <type 'str'> value repr: '+420721870631' <type 'str'> kontaktniOsobaPrijmeni value: <type 'str'> value repr: 'Kra\xc4\x8dmar' <type 'str'> nazevKontaktMistaPoruchy value: <type 'str'> value repr: 'Vr\xc3\xa1tnice' <type 'str'> psc value: <type 'str'> value repr: '46117' <type 'str'> mesto value: <type 'str'> value repr: 'Liberec' <type 'str'> geocodeAccuracy value: <type 'str'> value repr: 'Z' <type 'str'> adresa2 value: <type 'str'> value repr: '' <type 'str'> adresa1 value: <type 'str'> value repr: '17.listopadu 587/12' <type 'str'> zemDelka value: <type 'str'> value repr: '15.089062' <type 'str'> zemSirka value: <type 'str'> value repr: '50.770344' <type 'str'> elektrinaZaRokBezDph value: <type 'str'> value repr: '2052' <type 'str'> kontaktniOsobaJmeno value: <type 'str'> value repr: 'Zden\xc4\x9bk' <type 'str'> cisloSignysPartnera value: <type 'str'> value repr: '8' <type 'str'> telReseniPoruchy value: <type 'str'> value repr: '+420485355217' <type 'str'> nazevProdejnihoMista value: <type 'str'> value repr: 'Kolej LBC E' <type 'str'> dnuRokOtevreno value: <type 'str'> value repr: '340' <type 'str'> najemZaRokBezDph value: <type 'str'> value repr: '6000' ============ _SO_update start ===================== <type 'unicode'> tel_kontaktni_osoby value: <type 'str'> value repr: '+420721870631' <type 'unicode'> kontaktni_osoba_prijmeni value: <type 'str'> value repr: 'Kra\xc4\x8dmar' <type 'unicode'> nazev_kontakt_mista_poruchy value: <type 'str'> value repr: 'Vr\xc3\xa1tnice' <type 'unicode'> psc value: <type 'str'> value repr: '46117' <type 'unicode'> mesto value: <type 'str'> value repr: 'Liberec' <type 'unicode'> geocode_accuracy value: <type 'str'> value repr: 'Z' <type 'unicode'> adresa2 value: <type 'str'> value repr: '' <type 'unicode'> adresa1 value: <type 'str'> value repr: '17.listopadu 587/12' <type 'unicode'> zem_delka value: <type 'str'> value repr: '15.089062' <type 'unicode'> zem_sirka value: <type 'str'> value repr: '50.770344' <type 'unicode'> elektrina_za_rok_bez_dph value: <type 'str'> value repr: '2052' <type 'unicode'> kontaktni_osoba_jmeno value: <type 'str'> value repr: 'Zden\xc4\x9bk' <type 'unicode'> cislo_signys_partnera value: <type 'str'> value repr: '8' <type 'unicode'> tel_reseni_poruchy value: <type 'str'> value repr: '+420485355217' <type 'unicode'> nazev_prodejniho_mista value: <type 'str'> value repr: 'Kolej LBC E' <type 'unicode'> dnu_rok_otevreno value: <type 'str'> value repr: '340' <type 'unicode'> najem_za_rok_bez_dph value: <type 'str'> value repr: '6000' 2012-01-30 01:13:08,807 cherrypy.msg INFO HTTP: Page handler: <bound method ProdejniMista.ulozeniProdejnihoMista of <fantomas.controllers.ProdejniMista instance at 0x23fd638>> Traceback (most recent call last): File "/usr/lib/python2.5/site-packages/CherryPy-2.3.0-py2.5.egg/cherrypy/_cphttptools.py", line 121, in _run self.main() File "/usr/lib/python2.5/site-packages/CherryPy-2.3.0-py2.5.egg/cherrypy/_cphttptools.py", line 264, in main body = page_handler(*virtual_path, **self.params) File "<string>", line 3, in ulozeniProdejnihoMista File "/usr/local/lib/python2.5/site-packages/TurboGears-1.0.8-py2.5.egg/turbogears/controllers.py", line 360, in expose *args, **kw) File "<string>", line 5, in run_with_transaction File "/usr/local/lib/python2.5/site-packages/TurboGears-1.0.8-py2.5.egg/turbogears/database.py", line 359, in so_rwt retval = func(*args, **kw) File "<string>", line 5, in _expose File "/usr/local/lib/python2.5/site-packages/TurboGears-1.0.8-py2.5.egg/turbogears/controllers.py", line 373, in <lambda> mapping, fragment, args, kw))) File "/usr/local/lib/python2.5/site-packages/TurboGears-1.0.8-py2.5.egg/turbogears/controllers.py", line 410, in _execute_func output = errorhandling.try_call(func, *args, **kw) File "/usr/local/lib/python2.5/site-packages/TurboGears-1.0.8-py2.5.egg/turbogears/errorhandling.py", line 77, in try_call return func(self, *args, **kw) File "/TG_web/fantomas/controllers.py", line 317, in ulozeniProdejnihoMista prodMisto.set(**slovnikArgumentu) File "/usr/lib/python2.5/site-packages/SQLObject-0.10.1-py2.5.egg/sqlobject/main.py", line 1123, in set self._connection._SO_update(self, args) File "/usr/lib/python2.5/site-packages/SQLObject-0.10.1-py2.5.egg/sqlobject/dbconnection.py", line 509, in _SO_update myQuery = "UPDATE %s SET %s WHERE %s = (%s)" % (so.sqlmeta.table, ", ".join(["%s = (%s)" % (dbName, self.sqlrepr(value)) for dbName, value in values]), so.sqlmeta.idName, self.sqlrepr(so.id)) UnicodeDecodeError: 'ascii' codec can't decode byte 0xc4 in position 4: ordinal not in range(128) Petr |
From: Petr J. <pet...@tp...> - 2012-02-26 08:00:44
|
> > > sqlobject.dburi="firebird://user:password@localhost > > :3050/pathToTheDatabase/automaty.fdb?charset=utf8" > > > Finally, it looks I was focused on the column values to much, when hunting mentioned UnicodeDecodeError. You can find the solution here: The *names* list generated in: names = [self.sqlmeta.columns[v[0]].dbName for v in setters] (_SO_finishCreate method in main module) contains unicode type string elements (column names) for the columns, that are not manually defined in the model. because of that: return ("INSERT INTO %s (%s) VALUES (%s)" % (table, ', '.join(names), ', '.join([self.sqlrepr(v) for v in values]))) throws: UnicodeDecodeError "'ascii' codec can't decode byte 0xc4 in position 95: ordinal not in range(128)" Yes, it looks strange, but: 1. unicode type string is generated from the ', '.join(names), when there is at least one element in the list containing unicode type string 2. an unicode type string is generated from the "INSERT INTO %s (%s) VALUES (%s)" when the first part of the string formating is executed 3. when there is an encoded string in the values, it is not possible to execute the second part of the string formating and an exception ocures to understand it, try to experiment with the following code: table = "pokladna" myJoinedNames = u'id, autorizovano, vydal_osoby_id, mena_id, vydal_automaty_id, platny_zaznam, t_stamp, castka_czk, prijal_osoby_id, vycetky_od_automatu_id, prodejni_mista_id, prijal_automaty_id, pohyby_zbozi_id, castka, id_hash_vydal, poznamka, ucel_id, id_hash_prijal' myJoinedValues = "3219, 0, 3, 1, NULL, 1, '2012-02-26 00:41:31', 2000.0, 16, NULL, NULL, NULL, NULL, 2000.0, 3, '\xc4\x9b\xc5\xa1\xc4\x8d\xc5\x99', 3, NULL" print "INSERT INTO %s (%s) VALUES (%s)" % (table, myJoinedNames, myJoinedValues) As an "short and dirty" solution I have defined all columns in the affected table manually. I am not sure, if above mentioned is not caused because of I have rewritten columnsFromSchema and guessClass methods in the firebirdconnection module. I am pretty sure, Oleg will come with an other, elegant solution for this problem. Regards Petr |
From: Oleg B. <ph...@ph...> - 2012-02-26 11:26:24
|
On Sun, Feb 26, 2012 at 09:00:37AM +0100, Petr Jake?? wrote: > The *names* list generated in: > names = [self.sqlmeta.columns[v[0]].dbName for v in setters] > (_SO_finishCreate method in main module) > > contains unicode type string elements (column names) for the columns, that > are not manually defined in the model. The solution, I think, would be to fix columnsFromSchema to make column names strings, not unicode. Oleg. -- Oleg Broytman http://phdru.name/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Petr J. <pet...@tp...> - 2012-01-30 01:24:03
|
Finally some small success... It was necessary to define all the column names and column types manually in the table Class definition. Before inserting/updating to the database it is necessary to check, if the inserted value type is the same as declared in the table Class definition (it is not possible to send string in to the integer column for example). Strange. Is there some way how to check/set/convert column type according to the Class definition? I know, it is crazy, it is bypassing of the SQLObject functionality.... But I did not find an other solution for the communication with the Firebird database. Regards Petr |
From: Oleg B. <ph...@ph...> - 2012-01-30 09:33:53
|
On Mon, Jan 30, 2012 at 02:23:56AM +0100, Petr Jake?? wrote: > It was necessary to define all the column names and column types manually > in the table Class definition. That's ok. "fromDatabase" schema guessing in SQLObject is a bit primitive. It works slightly better for The Big Three (MySQL, Postgres, SQLite) but still far from ideal. You have a chance to improve the machinery for Firebird - it is in FirebirdConnection class, methods columnsFromSchema and guessClass. > Before inserting/updating to the database it is necessary to check, if the > inserted value type is the same as declared in the table Class definition That strange - it's the job of validators. > (it is not possible to send string in to the integer column for example). > Strange. Well, that's that job - IntCol's validator doesn't allow strings but allows any object that can be coerced to int. > Is there some way how to check/set/convert column type according to the > Class definition? See the code in .set() that converts input values: for name, value in kw.items(): from_python = getattr(self, '_SO_from_python_%s' % name, None) if from_python: dbValue = from_python(value, self._SO_validatorState) else: dbValue = value to_python = getattr(self, '_SO_to_python_%s' % name, None) if to_python: value = to_python(dbValue, self._SO_validatorState) Oleg. -- Oleg Broytman http://phdru.name/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Petr J. <pet...@tp...> - 2012-01-31 17:28:34
|
> That's ok. "fromDatabase" schema guessing in SQLObject is a bit > primitive. It works slightly better for The Big Three (MySQL, Postgres, > SQLite) but still far from ideal. > You have a chance to improve the machinery for Firebird - it is in > FirebirdConnection class, methods columnsFromSchema and guessClass. > With some luck I will check it this weekend (I can not promiss). It looks like it gives different results with different versions of Firebird. Actually I am trying to migrate TurboGears working application to an other machine. The Python, TurboGears, kinterbasdb, SQLObject are the same version. The only differences are version of Firebird and different Ubuntu version. > > > Before inserting/updating to the database it is necessary to check, if > the > > inserted value type is the same as declared in the table Class > definition > > That strange - it's the job of validators. > Yes, that is what I have found strange as well... > > > (it is not possible to send string in to the integer column for example). > > Strange. > > Well, that's that job - IntCol's validator doesn't allow strings but > allows any object that can be coerced to int. > > > Is there some way how to check/set/convert column type according to the > > Class definition? > > See the code in .set() that converts input values: > > for name, value in kw.items(): > from_python = getattr(self, '_SO_from_python_%s' % name, > None) > if from_python: > dbValue = from_python(value, self._SO_validatorState) > else: > dbValue = value > to_python = getattr(self, '_SO_to_python_%s' % name, None) > if to_python: > value = to_python(dbValue, self._SO_validatorState) > > I will check it. My feeling was it is not a problem to coerce str to int. OK, thanks a lot anyway. Petr |
From: Oleg B. <ph...@ph...> - 2012-01-31 19:17:05
|
Hi! On Tue, Jan 31, 2012 at 06:28:23PM +0100, Petr Jake?? wrote: > With some luck I will check it this weekend (I can not promiss). Eagerly waiting! > It looks > like it gives different results with different versions of Firebird. > Actually I am trying to migrate TurboGears working application to an other > machine. The Python, TurboGears, kinterbasdb, SQLObject are the same > version. The only differences are version of Firebird and different Ubuntu > version. Good luck resolving this! > I will check it. My feeling was it is not a problem to coerce str to int. In 0.13.0 validators becomes event stricter: * Validators became stricter: StringCol and UnicodeCol now accept only str, unicode or an instance of a class that implements __unicode__ (but not __str__ because every object has a __str__ method); BoolCol accepts only bool or int or an instance of a class that implements __nonzero__; IntCol accepts int, long or an instance of a class that implements __int__ or __long__; FloatCol accepts float, int, long or an instance of a class that implements __float__, __int__ or __long__. But you can add a validator/converter to any column yourself: from formencode.validators import Int class MyTable(SQLobject): column = IntCol(validator2=Int()) Oleg. -- Oleg Broytman http://phdru.name/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Petr J. <pet...@tp...> - 2012-02-23 18:41:06
|
On 31 January 2012 18:28, Petr Jakeš <pet...@tp...> wrote: > That's ok. "fromDatabase" schema guessing in SQLObject is a bit >> primitive. It works slightly better for The Big Three (MySQL, Postgres, >> SQLite) but still far from ideal. >> You have a chance to improve the machinery for Firebird - it is in >> FirebirdConnection class, methods columnsFromSchema and guessClass. >> > > With some luck I will check it this weekend (I can not promiss). It looks > like it gives different results with different versions of Firebird. > Actually I am trying to migrate TurboGears working application to an other > machine. The Python, TurboGears, kinterbasdb, SQLObject are the same > version. The only differences are version of Firebird and different Ubuntu > version. > After some investigation of the method columnsFromSchema it looks like Firebird (FB 2.5 SuperServer actualy) returns the information about the field type in following form: u'SHORT ' so it is necessary to add t = t.strip() in the for loop (tested) Anyway, I guess it cant hurt anything. HTH Petr |
From: Oleg B. <ph...@ph...> - 2012-02-23 20:06:04
|
On Thu, Feb 23, 2012 at 07:40:54PM +0100, Petr Jake?? wrote: > After some investigation of the method columnsFromSchema it looks like > Firebird (FB 2.5 SuperServer actualy) returns the information about the > field type in following form: > u'SHORT > ' > so it is necessary to add > t = t.strip() > in the for loop (tested) Ok, I'll add it. Oleg. -- Oleg Broytman http://phdru.name/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |