Re: [SQLObject] PostgreSQL warnings filling up the logs with PickleCol

[Tom C. <su...@ic...>]

Hey Oleg,

On Thursday 31 May 2012 16:16:41 Oleg Broytman wrote:
> Ok, found the problem - str() removes one level of backslashes so I have
> to double backslashes once again. See the attached patch - it passes all
> tests.

When I use the new patch on my test case, after adding some extended 
characters to the pickle column and setting the encoding to utf-8 (see below), 
i get this error:

------------------------
$ python pickleTest.py 
 1/QueryIns:  INSERT INTO server (id, config, name) VALUES (1, 
E'\\\\200\\\\002}q\\\\001(U\\\\005otherq\\\\002X\\\\010\\\\000\\\\000\\\\000f\\\\303\\\\266\\\\303\\\\266barq\\\\003U\\\\006remoteq\\\\004\\\\211U\\\\003dnsq\\\\005}q\\\\006(U\\\\006domainq\\\\007U\\\\007foo.barq\\\\010U\\\\010hostnameq\\\\011NuU\\\\010servicesq\\\\012]q\\\\013(U\\\\003webq\\\\014U\\\\010databaseq\\\\015eU\\\\006loggerq\\\\016NU\\\\005portsq\\\\017K\\\\002u.'::bytea, 
'foo')
 1/COMMIT  :  auto
 1/QueryOne:  SELECT name, config FROM server WHERE ((server.id) = (1))
 1/QueryR  :  SELECT name, config FROM server WHERE ((server.id) = (1))
 1/COMMIT  :  auto
Traceback (most recent call last):
  File "pickleTest.py", line 37, in <module>
    s1 = Server(name='foo')
  File "/usr/lib/pymodules/python2.6/sqlobject/main.py", line 1223, in 
__init__
    self._create(id, **kw)
  File "/usr/lib/pymodules/python2.6/sqlobject/main.py", line 1271, in _create
    self._SO_finishCreate(id)
  File "/usr/lib/pymodules/python2.6/sqlobject/main.py", line 1298, in 
_SO_finishCreate
    self._init(id)
  File "/usr/lib/pymodules/python2.6/sqlobject/main.py", line 936, in _init
    self._SO_selectInit(selectResults)
  File "/usr/lib/pymodules/python2.6/sqlobject/main.py", line 1154, in 
_SO_selectInit
    colValue = col.to_python(colValue, self._SO_validatorState)
  File "/usr/lib/pymodules/python2.6/formencode/api.py", line 413, in 
to_python
    value = tp(value, state)
  File "/usr/lib/pymodules/python2.6/formencode/compound.py", line 57, in 
_to_python
    to_python)
  File "/usr/lib/pymodules/python2.6/formencode/compound.py", line 118, in 
attempt_convert
    value = validate(validator, value, state)
  File "/usr/lib/pymodules/python2.6/formencode/compound.py", line 15, in 
to_python
    return validator.to_python(value, state)
  File "/usr/lib/pymodules/python2.6/sqlobject/col.py", line 1491, in 
to_python
    return pickle.loads(value)
cPickle.UnpicklingError: invalid load key, '\'.
------------------------

I also get these errors on my current app with the new patch.

With the previous patch both my app unit tests and the test app below runs 
fine.

I wish I had some more time to spend on this and try to debug the problem, but 
I am swamped at the moment :-(

For the time being I am using the first patch you sent on my production system 
and all seems fine so far.

If you can look into this further, it would be much appreciated, but 
otherwise, no problem. You have already been a great help!


Thanks,
 Tom


Updated test case with encoding and unicode pickle data:
----------------------------------------------------------------
# -*- coding: utf-8 -*-

from sqlobject import *

db = dict(user='tomc',
          passw='',
          host='',
          dbname='tomc')
dbURI = "postgres://%(user)s:%(passw)s@%(host)s/%(dbname)s" % db

connection = connectionForURI(dbURI)
sqlhub.processConnection = connection

# Default pickled data - try to use as many object types as possible for tests
defaultConfig = dict(ports=2,
                     remote=False,
                     logger=None,
                     services=['web', 'database'],
                     dns=dict(hostname=None,
                              domain='foo.bar'),
                     other=u'fööbar'
                     )

# Define the sampe table with a pickled column
class Server(SQLObject):
    name = StringCol()
    config = PickleCol(notNull=True, default=defaultConfig)

# Drop and create table
Server.dropTable(ifExists=True)
Server.createTable()

# Turn on SQLObejct debugging
connection.debug=True

# Add test records while tailing the postgresql log files
s1 = Server(name='foo')

----------------------------------------------------------------