sqlobject-discuss — All things SQLObject-related

Re: [SQLObject] Joins

[Ian B. <ia...@co...>]

Michel Thadeu wrote:
>>>>class Barrel(SQLObject):
> 
> ...   monkeys=MultipleJoin('Monkey')
> ...
> 
>>>>class Monkey(SQLObject):
> 
> ...   barrel=ForeignKey('Barrel')
> ...
> 
>>>>barrel=Barrel()
>>>>monkey=Monkey(barrel=barrel)
>>>>isinstance(barrel.monkeys, (list, tuple))
> 
> True
> 
> Why this is true?  Why can't the join return a select object, this way,
> if I need to count the monkeys of a barrel, I can use
> barrel.monkeys.count(), I can use sum() and other aggregate function I
> want... The actual way I need to rewrite the join to be able to apply
> these aggregate functions, or I use python to calculate what I need..

No good reason, except perhaps the original order that I implemented 
these features (complex select results after joins).

I'm also unhappy with select results, as they can be *too* lazy.  For 
instance, you can't test for truth value (to see if the query returns no 
rows).  I'm thinking that select results should be more stateful, but 
lazily so, caching the results.  This might help resolve some of the 
problems with databases that can't use multiple cursors -- if you make a 
query and there's another cursor waiting to finish its query (a lazy 
select iterator), it could quick finish before the other query was 
allowed to run, pushing its results into the SelectResult's cache.

It would be neat if, for instance, you could add to the query in of a 
select result.  Like barrel.monkeys.query(Monkey.q.species=='Spider'), 
which would be equivalent to Monkeys.select(
Monkey.q.barrel==barrel).query(Monkey.q.species=='Spider') , which would 
be equivalent to Monkey.select(AND(Monkey.q.barrel==barrel, 
Monkey.q.species='Spider')).

-- 
Ian Bicking  /  ia...@co...  /  http://blog.ianbicking.org

Re: [SQLObject] Problem with _init (docs could be clearer)

[Ian B. <ia...@co...>]

Carlos Ribeiro wrote:
> It's just a small issue...
> 
> The section "Initializing the objects" of the documentation mentions
> that, in order to customize the initialization code, one should
> override the _init method:

Noted and documentation corrected.

-- 
Ian Bicking  /  ia...@co...  /  http://blog.ianbicking.org

[SQLObject] Joins

[Michel T. <mic...@ya...>]

Hi people!

See this:

>>> class Barrel(SQLObject):
...   monkeys=MultipleJoin('Monkey')
...
>>> class Monkey(SQLObject):
...   barrel=ForeignKey('Barrel')
...
>>> barrel=Barrel()
>>> monkey=Monkey(barrel=barrel)
>>> isinstance(barrel.monkeys, (list, tuple))
True

Why this is true? Why can't the join return a select object, this way,
if I need to count the monkeys of a barrel, I can use
barrel.monkeys.count(), I can use sum() and other aggregate function I
want... The actual way I need to rewrite the join to be able to apply
these aggregate functions, or I use python to calculate what I need...

Is there a disvantage on using python to make the calculation? Does my
worry has foundation? Thanks for all help!


=====
--
Michel Thadeu Sabchuk
Curitiba/PR


	
	
		
_______________________________________________________ 
Yahoo! Acesso Grátis - Internet rápida e grátis. Instale o discador agora! http://br.acesso.yahoo.com/

[SQLObject] Problem with _init (docs could be clearer)

[Carlos R. <car...@gm...>]

It's just a small issue...

The section "Initializing the objects" of the documentation mentions
that, in order to customize the initialization code, one should
override the _init method:

"In general, you should not touch __init__. Instead use the _init
method, which is called after an object is fetched or inserted. This
method has the signature _init(self, id, connection=None,
selectResults=None), though you may just want to use _init(self,
*args, **kw)."

What the doc doesn't tell is that, if you write a new _init method,
you have to call the old one. Of course, one may say that this is
obvious, but I found it strange for two reasons:

-- I wrongly assumed that it was a hook for user customization, partly
due to the way the documentation is written. It mentions "override",
not "inherit". (we can debate for ages the meaning of "override" in
that context, but that was my interpretation).

-- I also interpreted the use of a generic signature as a sign that
the method really didn't care about what gets passed to it, and that I
could just use that information for my own business.

I suggest one simple ammendment to the documentation, just to make
clear that the standard _init has to be called, preferably _before_
one does anything else.

-- 
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: car...@gm...
mail: car...@ya...

[SQLObject] Inheritance

[Oleg B. <ph...@ph...>]

Hello!

   http://svn.colorstudy.com/home/phd/SQLObject/inheritance/

   I've fixed a minor problem with reST in docs/Inheritance.txt. Now I
am satisfied with the code, the doc, and the test.
   I am open for discussion.

Oleg.
-- 
     Oleg Broytmann            http://phd.pp.ru/            ph...@ph...
           Programmers don't die, they just GOSUB without RETURN.

Re: [SQLObject] Unicode support?

[Max I. <ma...@uc...>]

Ian Bicking wrote:

> So maybe Unicode is like dates, messy and ad hoc.  It might be better 
> (or worse) in database servers that accept basic data types; psycopg 
> does all its quoting on the client side.

I see. Amen.

> Also, if you are willing to use a default encoding, you could change 
> converters.StringLikeConverter:
> 
> def UnicodeConverter(value, db):
>     return StringLikeConverter(value.encode('utf-8'), db)
> registerConverter(unicode, UnicodeConverter)

Hmm. Looks like there are a lot of ways to solve this particular 
problem. The only complain is why they are missing in the docs. ;-)

Anyway, thanks a lot.

Surely have to look deeper into the sqlobject to find a sane replacement 
for my current approach.

As for "coverters", will this affect only db->python path or python->db 
path or both? Suspect the answer is UTSL. ;-)

I like very much the idea of being able to register some middle-man code 
between the python objects and db that could solve this conversion 
issues trasparently. I even have another use case at hand - make 
CurrencyCol understand my Money type.

The "validators" approach feels a bit out of place for this, as well as 
idea to subclass a particular *Col class.

> You'll still have to decode strings as they come out of the database, 
> but this will handle any queries you build.  Probably the current 
> behavior of using StringLikeConverter for unicode strings is bad, or at 
> least not helpful, because you'll get all sorts of errors if you have 
> any non-ASCII characters.

OK.

Re: [SQLObject] Re: Memory leak?

[Martin d'A. <Mar...@s2...>]

On Wed, 1 Dec 2004, Mike Thompson wrote:
> Ryan Harper wrote:
> > * Martin d'Anjou <Mar...@s2...> [2004-11-29 21:01]:
> >>Hi,
> >>
> >>I think I have found a memory leak. I have a loop that inserts hundreds of 
> > It is not a memory leak. I thought the same thing, but found out that
> > SQLObject caches object creation.  Hence, python's memory footprint
> > grows with each insert.
> Seems to be that this question and your response should be in the FAQ.

Along with the _connection.cache.clear() trick.

Martin

[SQLObject] Re: Memory leak?

[Mike T. <mik...@da...>]

Ryan Harper wrote:
> * Martin d'Anjou <Mar...@s2...> [2004-11-29 21:01]:
> 
>>Hi,
>>
>>I think I have found a memory leak. I have a loop that inserts hundreds of 
>>
> 
> 
> It is not a memory leak. I thought the same thing, but found out that
> SQLObject caches object creation.  Hence, python's memory footprint
> grows with each insert.
> 

Seems to be that this question and your response should be in the FAQ.

--
Mike

Re: [SQLObject] Memory leak?

[Ryan H. <rh...@sh...>]

* Leif K-Brooks <eu...@ec...> [2004-11-30 12:37]:
> Ryan Harper wrote:
> 
> >It would be nice to be able to toggle caching
> >
> You can, set the _cacheValues attribute to False. See 
> <URL:http://sqlobject.org/docs/SQLObject.html#sqlobject-class-specifying-your-class>.


from main.py:

# The _cacheValues attribute controls if you cache
# values fetched from the database.  We make sure
# it's set (default 1).

This controls whether SQLObject caches results, not whether it will
cache object creation.  A subtle distinction that isnt indicated in the
documentation.  In any case, I did set this value and observed
no reduction in memory footprint.

If it was unclear before, I would like to be able to toggle the
following:

1. caching of newly created objects (inserts)
2. Avoid issuing the subseqent select to initialize newly created
objects (_init() calls selectOne())

Ryan Harper

Re: [SQLObject] Memory leak?

[Martin d'A. <Mar...@s2...>]

Thanks for the informative reply!

> It would be nice to be able to toggle caching as well as instance _init.

I agree!

Martin

Re: [SQLObject] Unicode support?

[Ian B. <ia...@co...>]

Max Ischenko wrote:
>>> Most modern dbs and python drivers handle this problem transparently. 
>>> At least, that's my experience.
>>
>>    Can you show an example? Using a snippet of code, with a DB API
>> driver...
> 
> 
> Guess not. ;-)
> 
> Just checked with psycopg -- it does requre a parameter to be encoded 
> into something like utf-8.
> 
> Either my memory make me a disservice or this psycopg is somehow broken. 
> ;-)

I suspect it's something about Unicode in databases being a pain in the 
ass.  At least, that's what I'm guessing; I've never tried to do it, 
I've only stored ASCII and stuff that I treat as though its binary data.

I might note when I installed postgres on Debian, it asked me questions 
about encoding.  This might imply that encoding setup in an 
installation-wide (not per-database or per-session) fashion.  Then it 
also asked me about how I wanted to format my dates.  I answered ISO, 
but what madness would happen if someone selected US format dates?  I 
doubt psycopg knows anything about what format date the server is using. 
  Maybe these are just defaults, and by explicitly setting up the 
configuration for the connection you can avoid the madness.

So maybe Unicode is like dates, messy and ad hoc.  It might be better 
(or worse) in database servers that accept basic data types; psycopg 
does all its quoting on the client side.

Also, if you are willing to use a default encoding, you could change 
converters.StringLikeConverter:

def UnicodeConverter(value, db):
     return StringLikeConverter(value.encode('utf-8'), db)
registerConverter(unicode, UnicodeConverter)

You'll still have to decode strings as they come out of the database, 
but this will handle any queries you build.  Probably the current 
behavior of using StringLikeConverter for unicode strings is bad, or at 
least not helpful, because you'll get all sorts of errors if you have 
any non-ASCII characters.

-- 
Ian Bicking  /  ia...@co...  /  http://blog.ianbicking.org

Re: [SQLObject] Re: Postgres: DROP table CASCADE

[Ian B. <ia...@co...>]

Sidnei da Silva wrote:
> On Mon, Nov 29, 2004 at 09:47:37PM +0300, Oleg Broytmann wrote:
> | On Mon, Nov 29, 2004 at 12:31:31PM -0600, Ian Bicking wrote:
> | > python test.py -dpostgres
> | > it's not very flexible with its command-line arguments.
> | 
> |    I've made it much more flexible using getopt. Please look at main() at
> | 
> | http://svn.colorstudy.com/home/phd/SQLObject/inheritance/tests/test_sqlobject.py
> | 
> | PS. I renamed test.py to test_sqlobject.py to import from it.
> 
> What's wrong with optparse? Do we still need to support Python 2.2?

Yes, we should still support Python 2.2, but we could also include 
optparse/optik as part of the package.  We could either require people 
to install it (at least if they want run tests), or include it in the 
tarball and fit it into setup.py, or just include it in the source 
distribution (under like sqlobject/python24/optparse).  At some point we 
should be using the logging module as well, which raises the same issue.

-- 
Ian Bicking  /  ia...@co...  /  http://blog.ianbicking.org

Re: [SQLObject] Memory leak?

[Ryan H. <rh...@sh...>]

* Martin d'Anjou <Mar...@s2...> [2004-11-29 21:01]:
> Hi,
> 
> I think I have found a memory leak. I have a loop that inserts hundreds of 
>

It is not a memory leak. I thought the same thing, but found out that
SQLObject caches object creation.  Hence, python's memory footprint
grows with each insert.

in main.py:

def _SO_finishCreate(self, id=None):
   <snip>
   ...
   cache.created(id, self.__class__, self)

then in cache.py:

def created(self, id, obj):
        if self.doCache:
            self.cache[id] = obj
        else:
            self.expiredCache[id] = ref(obj)


You can flush the cache as follows:

table._connection.cache.clear()

In your example code:

def fill_table(size):
   for i in xrange(int(size)):
      p = Persons(name='Joe'+`i`)
      Persons._connection.cache.clear()

or you can delay until all inserts are done:

def fill_table(size):
   for i in xrange(int(size)):
      p = Persons(name='Joe'+`i`)
   Persons._connection.cache.clear()

On a related note, you will notice that in addition to the caching of
the object, immediately following the insert, SQLObject will issue a
select to fetch the values to create an instance in _init().  This 
degrades mysql insert performance.

It would be nice to be able to toggle caching as well as instance _init.

Ryan Harper

Re: [SQLObject] Unicode support?

[Oleg B. <ph...@ma...>]

On Tue, Nov 30, 2004 at 03:38:44PM +0200, Max Ischenko wrote:
> >class UnicodeStringValidator(validators.Validator):
> >
> >    def fromPython(self, value, state):
> >        return value.encode(db_encoding)
> >
> >    def toPython(self, value, state):
> >        return unicode(value, db_encoding)
> 
> What's the value of db_encoding here?

   Database encoding. I personnaly use koi8-r, but it can be anythying
your DB understands. If your DB understands UTF-8 (for ORDER BY,
UPPER()/LOWER() and all that) - go with it.

Oleg.
-- 
     Oleg Broytmann            http://phd.pp.ru/            ph...@ph...
           Programmers don't die, they just GOSUB without RETURN.

Re: [SQLObject] Unicode support?

[Max I. <ma...@uc...>]

Oleg Broytmann wrote:

>>   Subclass StrCol, and convert data from unicode to db encoding and
>>back in its validator.
> 
> 
> class UnicodeStringValidator(validators.Validator):
> 
>     def fromPython(self, value, state):
>         return value.encode(db_encoding)
> 
>     def toPython(self, value, state):
>         return unicode(value, db_encoding)

What's the value of db_encoding here?

Re: [SQLObject] Unicode support?

[Max I. <ma...@uc...>]

Oleg Broytmann wrote:

>>IIRC, native db drivers handle 
>>unicode input without problems.
 >
>    What is "unicode input"?

Parameters that are passed in as a unicode string (u'bla-bla-bla')

>>>In any case you cannot store unicode directly in the db
>>>- you must convert it to a db encoding.
>>
>>Most modern dbs and python drivers handle this problem transparently. At 
>>least, that's my experience.
>    Can you show an example? Using a snippet of code, with a DB API
> driver...

Guess not. ;-)

Just checked with psycopg -- it does requre a parameter to be encoded 
into something like utf-8.

Either my memory make me a disservice or this psycopg is somehow broken. ;-)

Re: [SQLObject] ._create() and .set()

[Oleg B. <ph...@ph...>]

On Mon, Nov 29, 2004 at 03:47:47PM -0600, Ian Bicking wrote:
> >   Well, there is a minor semantic difference. ._create() tests if all
> >passed keywords arguments are actually columns. .set() does not test it.
> >One can do self.set(noncolumn="value"), which results in assignment
> >self.noncolumn="value", but can't do ATable(noncolumn="value").
> >
> >   Should we change the semantic of .set(), so it does not allow
> >non-column keywords? In my opinion - yes, we should.
> 
> Yes, it's just an oversight that it doesn't check them.

   Commited. This is a big change, so I spent more time testing it. I
found I need a special treatment for inheritance.

PS. Also I made a minor optimization:

-        extra = dict(filter(f_not_column, kw.items()))
-        kw = dict(filter(f_is_column, kw.items()))
+        items = kw.items()
+        extra = dict(filter(f_not_column, items))
+        kw = dict(filter(f_is_column, items))

   From time to time I will do some code cleanup and optimizations.

Oleg.
-- 
     Oleg Broytmann            http://phd.pp.ru/            ph...@ph...
           Programmers don't die, they just GOSUB without RETURN.

Re: [SQLObject] Re: Postgres: DROP table CASCADE

[Oleg B. <ph...@ma...>]

On Tue, Nov 30, 2004 at 09:27:23AM -0200, Sidnei da Silva wrote:
> What's wrong with optparse?

   It's too verbose for such a small task.

   What's wrong with getopt? It does its job well.

> Do we still need to support Python 2.2?

   We need. Documentation states SQLObject support Python 2.2+. Python
2.2.3 is not so old yet, why broke support? Not all are living on the
bleeding edge.

Oleg.
-- 
     Oleg Broytmann            http://phd.pp.ru/            ph...@ph...
           Programmers don't die, they just GOSUB without RETURN.

Re: [SQLObject] OT: IP

[Oleg B. <ph...@ma...>]

On Tue, Nov 30, 2004 at 09:25:59AM -0200, Sidnei da Silva wrote:
> On Tue, Nov 30, 2004 at 01:57:49PM +0300, Oleg Broytmann wrote:
> | I've got a permission to publish snippets of code from our program.
> 
> Jeebus, If permission is required to publish that much lines of code,
> then something is severely broken in our world :(

   Very broken, indeed. Property, and especially intellectual property.
Copyrights, licences, patents, EULA, UCITA, DMCA...
   Unfortunately, noone has come with a solution acceptable to everyone.

Oleg.
-- 
     Oleg Broytmann            http://phd.pp.ru/            ph...@ph...
           Programmers don't die, they just GOSUB without RETURN.

Re: [SQLObject] Re: Postgres: DROP table CASCADE

[Sidnei da S. <si...@aw...>]

On Mon, Nov 29, 2004 at 09:47:37PM +0300, Oleg Broytmann wrote:
| On Mon, Nov 29, 2004 at 12:31:31PM -0600, Ian Bicking wrote:
| > python test.py -dpostgres
| > it's not very flexible with its command-line arguments.
| 
|    I've made it much more flexible using getopt. Please look at main() at
| 
| http://svn.colorstudy.com/home/phd/SQLObject/inheritance/tests/test_sqlobject.py
| 
| PS. I renamed test.py to test_sqlobject.py to import from it.

What's wrong with optparse? Do we still need to support Python 2.2?

-- 
Sidnei da Silva <si...@aw...>
http://awkly.org - dreamcatching :: making your dreams come true
http://www.enfoldsystems.com
http://plone.org/about/team#dreamcatcher

The meta-Turing test counts a thing as intelligent if it seeks to
devise and apply Turing tests to objects of its own creation.
		-- Lew Mammel, Jr.

Re: [SQLObject] Unicode support?

[Sidnei da S. <si...@aw...>]

On Tue, Nov 30, 2004 at 01:57:49PM +0300, Oleg Broytmann wrote:
| I've got a permission to publish snippets of code from our program.

Jeebus, If permission is required to publish that much lines of code,
then something is severely broken in our world :(

-- 
Sidnei da Silva <si...@aw...>
http://awkly.org - dreamcatching :: making your dreams come true
http://www.enfoldsystems.com
http://plone.org/about/team#dreamcatcher

<tenbytes> fag
<dash> this channel is made of LOVE AND PEACE!!
<tenbytes> oh

Re: [SQLObject] Unicode support?

[Oleg B. <ph...@ma...>]

On Tue, Nov 30, 2004 at 12:55:41PM +0200, Max Ischenko wrote:
> IIRC, native db drivers handle 
> unicode input without problems.

   What is "unicode input"?

> > In any case you cannot store unicode directly in the db
> > - you must convert it to a db encoding.
> 
> Most modern dbs and python drivers handle this problem transparently. At 
> least, that's my experience.

   Can you show an example? Using a snippet of code, with a DB API
driver...

Oleg.
-- 
     Oleg Broytmann            http://phd.pp.ru/            ph...@ph...
           Programmers don't die, they just GOSUB without RETURN.

Re: [SQLObject] Unicode support?

[Max I. <ma...@uc...>]

Oleg Broytmann wrote:
> 
>    Subclass StrCol, and convert data from unicode to db encoding and
> back in its validator.

May be.

>>Btw, Oleg's proposal about getting rid of sqlobject ad-hoc escaping can 
>>possibly solve the Unicode problem as well.
> 
> 
>    In what way?!

Well, I may be wrong here.

I didn't follow that thread in details, but AFAIU, SQLObject 
interpolates string by itself, probably using str(). And this would give 
a UnicodeError for non-ascii encodings. IIRC, native db drivers handle 
unicode input without problems. At least those I've used. And moreover, 
they return data in a record set in unicode as well.

 > In any case you cannot store unicode directly in the db
 > - you must convert it to a db encoding.

Most modern dbs and python drivers handle this problem transparently. At 
least, that's my experience. And discussed issues may be an indicator 
that the problem lies within the SQLObject itself.

--max.

Re: [SQLObject] Unicode support?

[Oleg B. <ph...@ma...>]

I've got a permission to publish snippets of code from our program.

On Tue, Nov 30, 2004 at 01:39:31PM +0300, Oleg Broytmann wrote:
>    Subclass StrCol, and convert data from unicode to db encoding and
> back in its validator.

class UnicodeStringValidator(validators.Validator):

    def fromPython(self, value, state):
        return value.encode(db_encoding)

    def toPython(self, value, state):
        return unicode(value, db_encoding)

stringValidator = UnicodeStringValidator()

class SOUnicodeStringCol(SOStringCol):

    def __init__(self, **kw):
        SOStringCol.__init__(self, **kw)
        self.validator = validators.All.join(stringValidator, self.validator)

class UnicodeStringCol(Col):
    baseClass = SOUnicodeStringCol

Oleg.
-- 
     Oleg Broytmann            http://phd.pp.ru/            ph...@ph...
           Programmers don't die, they just GOSUB without RETURN.

Re: [SQLObject] Unicode support?

[Oleg B. <ph...@ma...>]

On Tue, Nov 30, 2004 at 12:29:33PM +0200, Max Ischenko wrote:
> class User(SQLObject):
>     name = StringCol(length=96, default='')
> 
>     _get_name = getstring('name')
>     _set_name = setstring('name')
> 
> This solution provides convertion from Unicode (used throughout the 
> probram) to the utf-8 encoding accepted by SQLObject.
> 
> Obviously, this is far from ideal. Therefore I'd like to see how the 
> others deal with this problem.

   Subclass StrCol, and convert data from unicode to db encoding and
back in its validator.

> Btw, Oleg's proposal about getting rid of sqlobject ad-hoc escaping can 
> possibly solve the Unicode problem as well.

   In what way?! In any case you cannot store unicode directly in the db
- you must convert it to a db encoding.

Oleg.
-- 
     Oleg Broytmann            http://phd.pp.ru/            ph...@ph...
           Programmers don't die, they just GOSUB without RETURN.