Re: [Sqlrelay-discussion] Problem with Python cPickle and blobs
Brought to you by:
mused
Menu
▾
▴
Re: [Sqlrelay-discussion] Problem with Python cPickle and blobs
|
From: Ville S. <vs...@eb...> - 2005-07-28 08:29:37
|
Hi Dave,
thank you very much for putting your time into this. As I tried to explain
in my earlier mail (I'm not very good in explaining things!), the problem
happens when I use cPickle.dumps to generate a binary string, not a text
string. You get a binary string by specifying a second parameter '2'
(according to my source, Python Cookbook, 2nd ed):
bdata=cPickle.dumps(sdata, 2)
When I run your test script, it works, because cPickle.dumps(sdata)
generates a text string. But when I do the above change, I get an error:
{'spam': {'eggs': 'spamspam', 'answers': {1: '', 2: 'foo', 3: 'bar'}}}
Traceback (most recent call last):
File "./testsqlr.py", line 32, in ?
mydata=cPickle.loads(fields[0])
EOFError
It's not really necessary to use a binary string in my case, so I wouldn't
give this problem a very high priority. The Python Cookbook states that
using binary strings is "faster and takes up less space".
I haven't had time to try your suggestion about using getField() and
getFieldLength() for blobs. I'll try it in the next few days.
My SQLRelay is version 0.36 with rudiments 0.28.2.
Thanks again!
Ville
On Wed, 27 Jul 2005, David Muse wrote:
> Ville,
>
> I have run some tests with my development version and the following
> script works:
>
> #! /usr/bin/env python
>
> # Copyright (c) 2001 David Muse
> # See the file COPYING for more information.
>
> from SQLRelay import PySQLRClient
> import sys
> import string
> import cPickle
>
> con=PySQLRClient.sqlrconnection("localhost",8009,"/tmp/oracle8test.socket","oracle8test","oracle8test")
> #con.debugOn()
> cur=PySQLRClient.sqlrcursor(con)
>
> cur.sendQuery("drop table testtable");
> cur.sendQuery("create table testtable (col1 blob)")
>
> cur.prepareQuery("insert into testtable values(:val1)");
> sdata={'spam': {'eggs': 'spamspam', 'answers': {1:'', 2:'foo',
> 3:'bar'}}}
> bdata=cPickle.dumps(sdata)
> blen=len(bdata)
> cur.inputBindBlob('val1',bdata,blen)
> cur.executeQuery()
>
> cur.sendQuery("select * from testtable");
> mydata=cPickle.loads(cur.getField(0,0))
> print mydata
>
> cur.sendQuery("select * from testtable");
> fields=cur.getRow(0)
> mydata=cPickle.loads(fields[0])
> print mydata
>
> cur.sendQuery("drop table testtable");
>
>
> It prints out a tuple identical to the one sdata is set to. See if it
> doesn't work for you too. If not, then there may be a bug specific to
> the version of SQL Relay you're using. What version is that, by the
> way?
>
> Dave
>
>
> On Thu, 2005-07-21 at 16:10 +0100, Ville Silventoinen wrote:
>> Hi Dave,
>>
>> Python len seems to be working correctly with pickled data.
>>
>> I did find out that cPickle.dumps(sdata) works ok. This generates string
>> representation of the pickled data, which doesn't contain a \0 character.
>> (I had a typo in my earlier test, the create procedure used string data,
>> the modify procedure used binary, which is why I thought neither dumps
>> worked.)
>>
>> If I use the HIGHEST_PROTOCOL with dumps, it generates binary data. This
>> seems to have a problem with inputBindBlob(), unless I'm still doing
>> something wrong. So if I am right, you should be able to recreate the
>> problem with something like this:
>>
>> sdata={'spam': {'eggs': 'spamspam', 'answers': {1:'', 2:'foo', 3:'bar'}}}
>> ...
>> bdata = cPickle.dumps(sdata, 2) # uses HIGHEST_PROTOCOL
>> blen = len(bdata)
>> cur.inputBindBlob('colname', bdata, blen)
>> ...
>> # read the data by using normal select
>> ...
>> mydata = cPickle.loads(fields[0])
>> # this throws EOFError (at least in my case)
>>
>> I think '' generates \x00 or \x0, which becomes EOF with inputBindBlob.
>> Perhaps it encodes \x00 as \0? Or maybe the problem is in the select?
>> Is it ok to use normal select to read blobs?
>>
>> Thanks,
>> Ville
>>
>>
>> On Thu, 21 Jul 2005, Firstworks/4access wrote:
>>
>>> Ville,
>>>
>>> I was actually looking at that last night. I'm believe you're on track.
>>> inputBind() does a strlen() of the string to decide how many characters
>>> to store in the database. strlen() interprets \0's as end-of-string
>>> markers, so only part of the data gets stored in the blob. Then, later
>>> when you fetch and unpickle it, only part of it gets unpickled. I
>>> believe the solution is to use inputBindBlob() or inputBindClob() rather
>>> than inputBind() to store the pickled data. inputBindBlob() and
>>> inputBindClob() take a length parameter and ignore \0's.
>>>
>>> Give it a try and let me know whether it works or not.
>>>
>>> Dave
>>> dav...@fi...
>>>
>>> On Thu, 2005-07-21 at 12:31 +0100, Ville Silventoinen wrote:
>>>> Hi,
>>>>
>>>> I'm using PySQLRClient from sqlrelay-0.36 (with rudiments-0.28.2)
>>>> to store blobs in Oracle 9.2.0 database.
>>>>
>>>> I'm not sure if this has been fixed in the recent sqlrelay release,
>>>> but I'm experiencing problems when I store a Python data structure
>>>> (dictionary object) to a blob using Pythons cPickle module. Everything
>>>> works great until one of the values in the pickled object is an empty
>>>> string (''). After that cPickle.load fails to read the data, it raises
>>>> EOFError exception. So I wonder if PySQLRClient interprets the pickled
>>>> '' as an end-of-file character?
>>>>
>>>> If I try loading/saving similar data structure to a file, it works ok.
>>>>
>>>> Thanks for any advice,
>>>>
>>>> Ville
>>>>
>>>>
>>>>
>>>>
>>>> -------------------------------------------------------
>>>> SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
>>>> from IBM. Find simple to follow Roadmaps, straightforward articles,
>>>> informative Webcasts and more! Get everything you need to get up to
>>>> speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
>>>> _______________________________________________
>>>> Sqlrelay-discussion mailing list
>>>> Sql...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/sqlrelay-discussion
>>>>
>>>
>>>
>>>
>>> -------------------------------------------------------
>>> SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
>>> from IBM. Find simple to follow Roadmaps, straightforward articles,
>>> informative Webcasts and more! Get everything you need to get up to
>>> speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
>>> _______________________________________________
>>> Sqlrelay-discussion mailing list
>>> Sql...@li...
>>> https://lists.sourceforge.net/lists/listinfo/sqlrelay-discussion
>>>
>>
>>
>> -------------------------------------------------------
>> SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
>> from IBM. Find simple to follow Roadmaps, straightforward articles,
>> informative Webcasts and more! Get everything you need to get up to
>> speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
>> _______________________________________________
>> Sqlrelay-discussion mailing list
>> Sql...@li...
>> https://lists.sourceforge.net/lists/listinfo/sqlrelay-discussion
>>
>
>
>
> -------------------------------------------------------
> SF.Net email is Sponsored by the Better Software Conference & EXPO September
> 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices
> Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA
> Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf
> _______________________________________________
> Sqlrelay-discussion mailing list
> Sql...@li...
> https://lists.sourceforge.net/lists/listinfo/sqlrelay-discussion
>
|
