Re: [Sqlrelay-discussion] Problem with Python cPickle and blobs
Brought to you by:
mused
Menu
▾
▴
Re: [Sqlrelay-discussion] Problem with Python cPickle and blobs
|
From: David M. <dav...@fi...> - 2005-08-11 21:02:10
|
Yay! Glad it worked for you. 60% eh? definitely a nice improvement :)
Dave
On Thu, 2005-08-11 at 10:55 +0100, Ville Silventoinen wrote:
> Hi Dave,
>
> I finally had a chance to apply the CSQLRelay.C fix. It works!
> Using the "cPickle.dumps(sdata, 2)" speeds up my Python methods
> around 60%, which is a nice improvement.
>
> Thank you very much for the fix.
>
> Best regards,
> Ville
>
>
> On Thu, 4 Aug 2005, David Muse wrote:
>
> > Well, I didn't get to look at it that night. Sorry about that :)
> >
> > I figured it out. getField() works but fields returned from getRow()
> > don't. I fixed it though. If you replace src/api/python/CSQLRelay.C
> > with the attached file and rebuild/reinstall, it will work.
> >
> > Dave
> >
> > On Thu, 2005-07-28 at 11:02 -0400, Firstworks/4access wrote:
> >> Sorry, I forgot to mention. I tried cPickle.dumps(sdata) and
> >> cPickle.dumps(sdata,2) and both worked for me. I wonder if there's a
> >> bug in 0.36 that's causing the error. I'll give it a try tonight and
> >> let you know what I find.
> >>
> >> Dave
> >>
> >> On Thu, 2005-07-28 at 09:28 +0100, Ville Silventoinen wrote:
> >>> Hi Dave,
> >>>
> >>> thank you very much for putting your time into this. As I tried to explain
> >>> in my earlier mail (I'm not very good in explaining things!), the problem
> >>> happens when I use cPickle.dumps to generate a binary string, not a text
> >>> string. You get a binary string by specifying a second parameter '2'
> >>> (according to my source, Python Cookbook, 2nd ed):
> >>>
> >>> bdata=cPickle.dumps(sdata, 2)
> >>>
> >>> When I run your test script, it works, because cPickle.dumps(sdata)
> >>> generates a text string. But when I do the above change, I get an error:
> >>>
> >>> {'spam': {'eggs': 'spamspam', 'answers': {1: '', 2: 'foo', 3: 'bar'}}}
> >>> Traceback (most recent call last):
> >>> File "./testsqlr.py", line 32, in ?
> >>> mydata=cPickle.loads(fields[0])
> >>> EOFError
> >>>
> >>> It's not really necessary to use a binary string in my case, so I wouldn't
> >>> give this problem a very high priority. The Python Cookbook states that
> >>> using binary strings is "faster and takes up less space".
> >>>
> >>> I haven't had time to try your suggestion about using getField() and
> >>> getFieldLength() for blobs. I'll try it in the next few days.
> >>>
> >>> My SQLRelay is version 0.36 with rudiments 0.28.2.
> >>>
> >>> Thanks again!
> >>>
> >>> Ville
> >>>
> >>>
> >>> On Wed, 27 Jul 2005, David Muse wrote:
> >>>
> >>>> Ville,
> >>>>
> >>>> I have run some tests with my development version and the following
> >>>> script works:
> >>>>
> >>>> #! /usr/bin/env python
> >>>>
> >>>> # Copyright (c) 2001 David Muse
> >>>> # See the file COPYING for more information.
> >>>>
> >>>> from SQLRelay import PySQLRClient
> >>>> import sys
> >>>> import string
> >>>> import cPickle
> >>>>
> >>>> con=PySQLRClient.sqlrconnection("localhost",8009,"/tmp/oracle8test.socket","oracle8test","oracle8test")
> >>>> #con.debugOn()
> >>>> cur=PySQLRClient.sqlrcursor(con)
> >>>>
> >>>> cur.sendQuery("drop table testtable");
> >>>> cur.sendQuery("create table testtable (col1 blob)")
> >>>>
> >>>> cur.prepareQuery("insert into testtable values(:val1)");
> >>>> sdata={'spam': {'eggs': 'spamspam', 'answers': {1:'', 2:'foo',
> >>>> 3:'bar'}}}
> >>>> bdata=cPickle.dumps(sdata)
> >>>> blen=len(bdata)
> >>>> cur.inputBindBlob('val1',bdata,blen)
> >>>> cur.executeQuery()
> >>>>
> >>>> cur.sendQuery("select * from testtable");
> >>>> mydata=cPickle.loads(cur.getField(0,0))
> >>>> print mydata
> >>>>
> >>>> cur.sendQuery("select * from testtable");
> >>>> fields=cur.getRow(0)
> >>>> mydata=cPickle.loads(fields[0])
> >>>> print mydata
> >>>>
> >>>> cur.sendQuery("drop table testtable");
> >>>>
> >>>>
> >>>> It prints out a tuple identical to the one sdata is set to. See if it
> >>>> doesn't work for you too. If not, then there may be a bug specific to
> >>>> the version of SQL Relay you're using. What version is that, by the
> >>>> way?
> >>>>
> >>>> Dave
> >>>>
> >>>>
> >>>> On Thu, 2005-07-21 at 16:10 +0100, Ville Silventoinen wrote:
> >>>>> Hi Dave,
> >>>>>
> >>>>> Python len seems to be working correctly with pickled data.
> >>>>>
> >>>>> I did find out that cPickle.dumps(sdata) works ok. This generates string
> >>>>> representation of the pickled data, which doesn't contain a \0 character.
> >>>>> (I had a typo in my earlier test, the create procedure used string data,
> >>>>> the modify procedure used binary, which is why I thought neither dumps
> >>>>> worked.)
> >>>>>
> >>>>> If I use the HIGHEST_PROTOCOL with dumps, it generates binary data. This
> >>>>> seems to have a problem with inputBindBlob(), unless I'm still doing
> >>>>> something wrong. So if I am right, you should be able to recreate the
> >>>>> problem with something like this:
> >>>>>
> >>>>> sdata={'spam': {'eggs': 'spamspam', 'answers': {1:'', 2:'foo', 3:'bar'}}}
> >>>>> ...
> >>>>> bdata = cPickle.dumps(sdata, 2) # uses HIGHEST_PROTOCOL
> >>>>> blen = len(bdata)
> >>>>> cur.inputBindBlob('colname', bdata, blen)
> >>>>> ...
> >>>>> # read the data by using normal select
> >>>>> ...
> >>>>> mydata = cPickle.loads(fields[0])
> >>>>> # this throws EOFError (at least in my case)
> >>>>>
> >>>>> I think '' generates \x00 or \x0, which becomes EOF with inputBindBlob.
> >>>>> Perhaps it encodes \x00 as \0? Or maybe the problem is in the select?
> >>>>> Is it ok to use normal select to read blobs?
> >>>>>
> >>>>> Thanks,
> >>>>> Ville
> >>>>>
> >>>>>
> >>>>> On Thu, 21 Jul 2005, Firstworks/4access wrote:
> >>>>>
> >>>>>> Ville,
> >>>>>>
> >>>>>> I was actually looking at that last night. I'm believe you're on track.
> >>>>>> inputBind() does a strlen() of the string to decide how many characters
> >>>>>> to store in the database. strlen() interprets \0's as end-of-string
> >>>>>> markers, so only part of the data gets stored in the blob. Then, later
> >>>>>> when you fetch and unpickle it, only part of it gets unpickled. I
> >>>>>> believe the solution is to use inputBindBlob() or inputBindClob() rather
> >>>>>> than inputBind() to store the pickled data. inputBindBlob() and
> >>>>>> inputBindClob() take a length parameter and ignore \0's.
> >>>>>>
> >>>>>> Give it a try and let me know whether it works or not.
> >>>>>>
> >>>>>> Dave
> >>>>>> dav...@fi...
> >>>>>>
> >>>>>> On Thu, 2005-07-21 at 12:31 +0100, Ville Silventoinen wrote:
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> I'm using PySQLRClient from sqlrelay-0.36 (with rudiments-0.28.2)
> >>>>>>> to store blobs in Oracle 9.2.0 database.
> >>>>>>>
> >>>>>>> I'm not sure if this has been fixed in the recent sqlrelay release,
> >>>>>>> but I'm experiencing problems when I store a Python data structure
> >>>>>>> (dictionary object) to a blob using Pythons cPickle module. Everything
> >>>>>>> works great until one of the values in the pickled object is an empty
> >>>>>>> string (''). After that cPickle.load fails to read the data, it raises
> >>>>>>> EOFError exception. So I wonder if PySQLRClient interprets the pickled
> >>>>>>> '' as an end-of-file character?
> >>>>>>>
> >>>>>>> If I try loading/saving similar data structure to a file, it works ok.
> >>>>>>>
> >>>>>>> Thanks for any advice,
> >>>>>>>
> >>>>>>> Ville
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> -------------------------------------------------------
> >>>>>>> SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
> >>>>>>> from IBM. Find simple to follow Roadmaps, straightforward articles,
> >>>>>>> informative Webcasts and more! Get everything you need to get up to
> >>>>>>> speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
> >>>>>>> _______________________________________________
> >>>>>>> Sqlrelay-discussion mailing list
> >>>>>>> Sql...@li...
> >>>>>>> https://lists.sourceforge.net/lists/listinfo/sqlrelay-discussion
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> -------------------------------------------------------
> >>>>>> SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
> >>>>>> from IBM. Find simple to follow Roadmaps, straightforward articles,
> >>>>>> informative Webcasts and more! Get everything you need to get up to
> >>>>>> speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
> >>>>>> _______________________________________________
> >>>>>> Sqlrelay-discussion mailing list
> >>>>>> Sql...@li...
> >>>>>> https://lists.sourceforge.net/lists/listinfo/sqlrelay-discussion
> >>>>>>
> >>>>>
> >>>>>
> >>>>> -------------------------------------------------------
> >>>>> SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
> >>>>> from IBM. Find simple to follow Roadmaps, straightforward articles,
> >>>>> informative Webcasts and more! Get everything you need to get up to
> >>>>> speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
> >>>>> _______________________________________________
> >>>>> Sqlrelay-discussion mailing list
> >>>>> Sql...@li...
> >>>>> https://lists.sourceforge.net/lists/listinfo/sqlrelay-discussion
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> -------------------------------------------------------
> >>>> SF.Net email is Sponsored by the Better Software Conference & EXPO September
> >>>> 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices
> >>>> Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA
> >>>> Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf
> >>>> _______________________________________________
> >>>> Sqlrelay-discussion mailing list
> >>>> Sql...@li...
> >>>> https://lists.sourceforge.net/lists/listinfo/sqlrelay-discussion
> >>>>
> >>>
> >>>
> >>> -------------------------------------------------------
> >>> SF.Net email is Sponsored by the Better Software Conference & EXPO September
> >>> 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices
> >>> Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA
> >>> Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf
> >>> _______________________________________________
> >>> Sqlrelay-discussion mailing list
> >>> Sql...@li...
> >>> https://lists.sourceforge.net/lists/listinfo/sqlrelay-discussion
> >>>
> >>
> >>
> >>
> >> -------------------------------------------------------
> >> SF.Net email is Sponsored by the Better Software Conference & EXPO September
> >> 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices
> >> Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA
> >> Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf
> >> _______________________________________________
> >> Sqlrelay-discussion mailing list
> >> Sql...@li...
> >> https://lists.sourceforge.net/lists/listinfo/sqlrelay-discussion
> >>
> >
>
>
> -------------------------------------------------------
> SF.Net email is Sponsored by the Better Software Conference & EXPO
> September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices
> Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA
> Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf
> _______________________________________________
> Sqlrelay-discussion mailing list
> Sql...@li...
> https://lists.sourceforge.net/lists/listinfo/sqlrelay-discussion
>
|
