sqlobject-discuss — All things SQLObject-related

Re: [SQLObject] thoughts on structure and components

[Bud P. B. <bu...@si...>]

Hi Ian,

many thanks for your feedback.  I respond below..


On 08 May 2003 22:13:51 -0500
Ian Bicking <ia...@co...> wrote:

> I feel confident that foreign keys are hard, and I'm okay with that. 
> Which is to say -- the interface could use improvement (e.g., remove the
> double defining of the foreign key and the external join), but I don't
> think there's deep problems.

I'm experimenting a bit with my thoughts and yet have to figure out
whether my intuitions verify.  If I get some code to work, I'll send
it and we'll talk about s.th. more concrete.  What triggered my
thinking was a difficulty that didn't match _my_ intuition--so maybe
it reveals more about my thinking than it describes SQLObject..


> > Application Object:
> >   An Application Object is independent of backend storage
> 
> Independent of "backend storage", or independent of any particular
> backend storage that fits the rest of the model?
> 
> I.e., independent like SQLObject currently is (by changing _connection),
> or something more (like, doesn't inherit from SQLObject)...?

In my thought experiment, I thought it would be nice to write a class
without worries about persistance--just the business logic.  In a
second step (in my line of thought using inheritance), this is made
persistant.  This approach would make it possible to easily eachange
persistance layers... But it is really more of a devide and conquer
motivation...  Not sure whether that is possible--but think for
example of pre-existing classes that you want to make persistent..


> > Storable:
> >   A superclass that makes application objects storable.  
> >   It adds methods that interact with the Backend in order to
> >   - store
> >   - update
> >   - delete
> >   the object
> >   In addition, it provided methods to easily access related objects
> 
> I can see how this might be stylistically different from the SQLObject
> class, but how is it otherwise different?  (Well, there's SQLObject and
> MetaSQLObject, but those are tightly coupled, and I think justifiably
> so).

Maybe this is not at all different from SQLObject???  Can I write a
class and then make what I call a storable application class that
inherits from both the business-logic-only class and SQLObject?


> SQLObject is tightly coupled to DBConnection (i.e., they chat a lot,
> there's no well defined interface between the two, and information
> that's available between the two of them isn't necessarily otherwise
> public).  I can imagine them being decoupled, but only after SQLObject
> had a long time to mature.

I believe that my thoughts kind of split up DBConnection into several
different entities (connection, dbdriver, and maybe also backend?).
This way, I can for example define what I call mappings before opening
an actual connection to the dbms.  I see this as a sort of decoupling
of components..

> Now, SQLObject does place some restrictions on how you can create your
> classes, and some (though less) restrictions on the backend (the backend
> is less restricted because an RDBMS is already a much more restricted
> environment than Python).
> 
> But SQLObject is *not* an OODBMS (with RDBMS backends).  It's not ZODB
> or any of those... they use the RDBMS as a backend, where in SQLObject
> the Python class is more-or-less the slave to the RDBMS model and
> backend.  

I believe I think along the same lines.  I don't attempt to hide the
RDBMS issues as an oodb would nor have I (so far) thought of hiding
storage operations...  (Again, I describe my thoughts and don't
necessarily relate to SQLObject)

I believe compared to SQLObject (which is difficult at my current
vagueness of thought), I may have an additional abstraction layer
between objects and the db, represented by the Attr class that
implements my "higher-level" physical types.  

> There's a strong conceptual tie as well, which is a large
> benefit -- even if you let SQLObject create you schema, and don't touch
> the database except through SQLObject, you are using relational
> metaphors, which many people are comfortable with (and for more reasons
> than familiarity).
> 
> Inheritance falls into this, IMHO.  You can't support Python inheritance
> in an RDBMS.  

Actually, some RDBMS such as PostgreSQL support inheritance.   But I
haven't thought of implementing inheritance much.  I don't think along
the lines of an oodbms such as Zope DB...

> You can support something that vaguely approximates Python
> inheritance, and can solve some of the same problems, but it's not the
> same thing.

I guess you're right.  Haven't thought about this in detail.  

> But maybe you are really thinking of a super-superclass, i.e., something
> that SQLObject would inherit from, as well as other significantly
> different kinds of storage mechanisms.

I don't think so, but I don't understand the super-super concept
well...

> > Backend:
> >   An object that manages the interaction with the actual database.
> >   It has the following components:
> >   - mapping
> >   - dbDriver
> >   - dbConnection
> >   A first version can live with a single driver and connection--a later
> >   version may be able to work with multiple databases
> > 
> > Connection:
> >   An actual connection or connection pool that interacts with the dbms
> > 
> > DBdriver:
> >   A class that provides functionality that depends on the kind of connection
> >   used, typically depending on:
> >   - dbms used (e.g., postgres)
> >   - DBI module (and version)
> >   The DBdriver supports a subset of dbCols
> 
> BConnection is something of a misnomer -- DBDriver would probably be a
> better name (especially for the poorly named DBMConnection, which
> doesn't connect to anything).  Well, in the beginning DBConnection was
> really just a primitive form of connection pool, but that part is
> relatively small now.  There's nothing particularly important about the
> way they are joined.  If they weren't, though, the interface would look
> pretty much the same (since the type of connection you make implies
> which DBDriver you would use).  Separating DBConnection from DBDriver
> isn't a very significant change, since they are naturally coupled.

I separated the two since it makes it possible to define mappings
before having the connection in place.  Trying to devide and conquer,
I tried to separate the creation of SQL (strings) and actually
communicating with the dmbs..

> Now, there is the possibility of different kinds of connection pooling
> and general connection usage, but that's orthogonal to both connections
> themselves, and the drivers for those connections.
> 
> > Mapping:
> >   Description of the physical representation in a dbms-independent way.  
> >   The mapping is described by the following components:
> >   - Attributes of Storable Objects
> >   - Relationships of Storable Objects
> 
> Now, if I'm reading it right, this would be the most significant
> difference with SQLObject -- a mapping object is a separate from the
> class.  But why?  

Yes, I believe this is probably the most significant change--probably
at the core of what I call "weak coupling".  A mapping just
_describes_ the storage decisions made, doing nothing else at all.
While decoupling was my prime motivation for thinking this way, it may
also make it easier to cleanly separate representation decisions from
the rest of the code, making it easier to modify, tune, etc.

> Hmm... on second thought I see a reason that while you
> would potentially define fields in the class, you may wish to define the
> way those map to a database separately.  Then the class wouldn't be tied
> so much to SQLObject -- a very different kind of mapping could be used
> to take the same class and store it over SOAP.

Exactly

> I still think the current definitions would be easier and clearer for
> most uses.  At some point you may put it on a different backend, that
> may have different database names (for instance).

Just my intuition, that could be wrong--but I think whether to
separate or not has very little impact on the ease of defining the
representation decisions.  Something to verify..

My approach basically follows a pattern (maybe not in a formal sense)
that I have used in other projects of mine:  to represent just
decisions as objects and separate how to act on those decisions in
different classes..

> Right now you could hack something together to do this by using a Style
> subclass that fed off a mapping table (instead of using algorithms to
> map names).

This surprises me.  I see style and mapping as reasonably independent
things.  Style just defines some defaults (such as converting names
form case sensitive to case insensitive) and if I chose an appropriate
style, it saves me some typing.  

> But maybe I'm not clear on what you envision for mapping, since I would
> expect it would belong to the class 

I believe that here we think differently.  I explicitly want the
mapping to live independently of the class.  Maybe overly simplified,
an object asks the backend to store it, the backend matches a mapping
with the class of the object, asks the dbdriver to produce the
appropriate sql (passing it mapping and object) and asking the
connection to send it off...  But the essence of decoupling is that
the mapping is independent of the application class itself.  

Luke recently pointed to some Ambler articles and maybe my thoughts
are influenced by that.  The architecture Ambler describes actually
keeps the mappings in an rdbms and offers an administration interface
for mappings.  Obviously, I don't attempt to go that far..

> (with the class having N mappings
> for N backends).  I don't see how it would be part of the backend, so
> maybe you have something else in mind.

The mapping is mostly independent of the dmbs used.  So I don't see a
need to keep multiple mappings for a given class for different
backends.  Backend, in my understanding of the concept, is a class
that brings together some components (mapping, connection, dbdriver,
etc) and knows how to make them play together.  It provides the single
interface to storable objects.  A given object class will alway relate
to a single backend;  different classes may use different backends
since, for example, their data resides in different dmbs...

I suppose my quick writeup wasn't very clear... (bad, but difficult to
avoid...)

> > DbCol:  
> >   A low level physical type to represent a single field in the dbms. It is
> >   an abstraction of the actual SQL type since different DBdrivers can map it
> >   to different SQL expressions.  The types are named closely to standard
> >   SQL. Types such as TIMESTAMP could for example be translated to DateTime
> >   by a mySQL dbDriver.  
> 
> Pretty much Col...

Yes, exactly.  


> > Attribute:
> >   A high-level physical type to represent the higher-level physical rep. 
> >   This includes marshalling to and from a lower-level representation. 
> >   Marshalling can do things such as encoding structured objects (phone
> >   number with country and area code) to strings, pickling, mapping a single
> >   attribute (a point) to multiple columns (x, y or lat,long), etc.  
> 
> And something like this will come around at some point, it's just not
> developed yet.

Great!  So I hope my thoughts can help shape this...

I have a little newer version of my class diagram that I attach in
case you are interested in following the progress of my thoughts...

thanks for the interesting discussion

cheers

--b

Re: [SQLObject] thoughts on structure and components

[Ian B. <ia...@co...>]

Well, some random thoughts (composed bottom-up, if that helps it make
sense)

On Thu, 2003-05-08 at 04:37, Bud P.Bruegger wrote:
> Hi everyone,
> 
> Been thinking these days and would like to share some of what I came
> up with.  I suppose this could be of interest just for academic
> purposes or for a possible future refactoring of SQLObject.  If it's
> not of interest, just ignore.  
> 
> My thinking was triggered by:
> - My wish for higher-level physical types that users construct
>   depending on the application
> - My difficulties to easily add the things to SQLObject that I
>   expected to be easy:  e.g., foreign keys...

I feel confident that foreign keys are hard, and I'm okay with that. 
Which is to say -- the interface could use improvement (e.g., remove the
double defining of the foreign key and the external join), but I don't
think there's deep problems.

> - I have an intuition (that I cannot materialize) that there is some
>   strong coupling in SqlObject that steepens the learning curve and
>   makes some behavior unintuitive.  It may be that also the issues of
>   difficulties of subclassing etc. are falling in this area...  As I
>   said, I have no overly clear ideas to offer.  
> 
> So my first product out of this is just a clarification of what
> concepts (classes) there should be and how they relate with each
> other.  A first rough writeup of my ideas follows here:
> 
> Concepts:
> ---------
> 
> Application Object:
>   An Application Object is independent of backend storage

Independent of "backend storage", or independent of any particular
backend storage that fits the rest of the model?

I.e., independent like SQLObject currently is (by changing _connection),
or something more (like, doesn't inherit from SQLObject)...?

> Storable:
>   A superclass that makes application objects storable.  
>   It adds methods that interact with the Backend in order to
>   - store
>   - update
>   - delete
>   the object
>   In addition, it provided methods to easily access related objects

I can see how this might be stylistically different from the SQLObject
class, but how is it otherwise different?  (Well, there's SQLObject and
MetaSQLObject, but those are tightly coupled, and I think justifiably
so).

SQLObject is tightly coupled to DBConnection (i.e., they chat a lot,
there's no well defined interface between the two, and information
that's available between the two of them isn't necessarily otherwise
public).  I can imagine them being decoupled, but only after SQLObject
had a long time to mature.

Now, SQLObject does place some restrictions on how you can create your
classes, and some (though less) restrictions on the backend (the backend
is less restricted because an RDBMS is already a much more restricted
environment than Python).

But SQLObject is *not* an OODBMS (with RDBMS backends).  It's not ZODB
or any of those... they use the RDBMS as a backend, where in SQLObject
the Python class is more-or-less the slave to the RDBMS model and
backend.  There's a strong conceptual tie as well, which is a large
benefit -- even if you let SQLObject create you schema, and don't touch
the database except through SQLObject, you are using relational
metaphors, which many people are comfortable with (and for more reasons
than familiarity).

Inheritance falls into this, IMHO.  You can't support Python inheritance
in an RDBMS.  You can support something that vaguely approximates Python
inheritance, and can solve some of the same problems, but it's not the
same thing.

But maybe you are really thinking of a super-superclass, i.e., something
that SQLObject would inherit from, as well as other significantly
different kinds of storage mechanisms.

> Backend:
>   An object that manages the interaction with the actual database.
>   It has the following components:
>   - mapping
>   - dbDriver
>   - dbConnection
>   A first version can live with a single driver and connection--a later
>   version may be able to work with multiple databases
> 
> Connection:
>   An actual connection or connection pool that interacts with the dbms
> 
> DBdriver:
>   A class that provides functionality that depends on the kind of connection
>   used, typically depending on:
>   - dbms used (e.g., postgres)
>   - DBI module (and version)
>   The DBdriver supports a subset of dbCols

BConnection is something of a misnomer -- DBDriver would probably be a
better name (especially for the poorly named DBMConnection, which
doesn't connect to anything).  Well, in the beginning DBConnection was
really just a primitive form of connection pool, but that part is
relatively small now.  There's nothing particularly important about the
way they are joined.  If they weren't, though, the interface would look
pretty much the same (since the type of connection you make implies
which DBDriver you would use).  Separating DBConnection from DBDriver
isn't a very significant change, since they are naturally coupled.

Now, there is the possibility of different kinds of connection pooling
and general connection usage, but that's orthogonal to both connections
themselves, and the drivers for those connections.

> Mapping:
>   Description of the physical representation in a dbms-independent way.  
>   The mapping is described by the following components:
>   - Attributes of Storable Objects
>   - Relationships of Storable Objects

Now, if I'm reading it right, this would be the most significant
difference with SQLObject -- a mapping object is a separate from the
class.  But why?  Hmm... on second thought I see a reason that while you
would potentially define fields in the class, you may wish to define the
way those map to a database separately.  Then the class wouldn't be tied
so much to SQLObject -- a very different kind of mapping could be used
to take the same class and store it over SOAP.

I still think the current definitions would be easier and clearer for
most uses.  At some point you may put it on a different backend, that
may have different database names (for instance).

Right now you could hack something together to do this by using a Style
subclass that fed off a mapping table (instead of using algorithms to
map names).

But maybe I'm not clear on what you envision for mapping, since I would
expect it would belong to the class (with the class having N mappings
for N backends).  I don't see how it would be part of the backend, so
maybe you have something else in mind.

> DbCol:  
>   A low level physical type to represent a single field in the dbms. It is an
>   abstraction of the actual SQL type since different DBdrivers can map it to
>   different SQL expressions.  The types are named closely to standard SQL.
>   Types such as TIMESTAMP could for example be translated to DateTime by a
>   mySQL dbDriver.  

Pretty much Col...

> Attribute:
>   A high-level physical type to represent the higher-level physical rep.  This
>   includes marshalling to and from a lower-level representation.  Marshalling
>   can do things such as encoding structured objects (phone number with country
>   and area code) to strings, pickling, mapping a single attribute (a point) to
>   multiple columns (x, y or lat,long), etc.  

And something like this will come around at some point, it's just not
developed yet.

  Ian

Re: [SQLObject] cacheValues, destroySelf, etc

[Ian B. <ia...@co...>]

On Thu, 2003-05-08 at 15:42, Luke Opperman wrote:
> because _SO_autoInitDone is never (anywhere in SQLObject.py) set to
> true. Now, I can't tell if this is just a missing line somewhere, or
> if autoInitDone is an old flag that doesn't have any meaning now that
> we're using _SO_selectInit. Ian?
> 
> For now, my best-guess solution has been to replace the few instances
> of _SO_autoInitDone with _cacheValues where appropriate, to mirror
> the setup in addColumn. Seems to be working so far...

Yes, that was obsolete code hanging around and causing problems, and I
took it out of CVS a couple days ago.

> 2. Is there a good reason SQLObject.destroy(self) was renamed to
> SQLObject.destroySelf(self)? It just doesn't read correctly to me,
> and breaks some other code we have, and.. I just don't get it. Unless
> the intention was to eventually have a cascading delete on the python
> side. (we just enforce such things through Postgres, but maybe
> there's a case for a having it on the python side.) In which case I'd
> still argue to leave .destroy() as the default (since cascade may be
> enforced on the db side), and either have an argument 'cascade', or a
> separate method .destroyRelated() or similar.

I've been planning for destroySelf to delete rows from RelatedJoins, but
further cascading is up in the air.  Maybe the join objects could
indicate how cascading should work (so it would be in the class
definition, not in the method invocation).

But that wasn't really the reason for the change.  I added a "delete"
class method, and delete and destroy are like two ways of saying the
same thing.  Plus I don't like the destroy method (whatever its name),
and I was thinking of taking it out (in favor of the delete class
method).  Self-destructing objects are weird, and they don't fit the
Python model... though at some level that has to happen because the
underlying databases aren't like Python (with garbage collecting, etc). 
I.e., instances have to become obsolete even before they are garbage
collected.

Maybe selfDestruct() would be a better name :)

  Ian

[SQLObject] cacheValues, destroySelf, etc

[Luke O. <lu...@me...>]

Hi all -

Ok, I've finally had some time to put towards SQLObject-based
Transactions (backend-independent), as a test implementation. I'm
doing it with an in-memory connection class that is swapped for the
real object's connection, and then replaced. Seems to be working
pretty well. But I'm having a few unrelated issues that are getting
in the way of finalizing this.

This is all based on CVS from a week or so ago, but for some reason I
can't get latest CVS to try on right now.

1. cacheValues. Now that I'm poking around with modifying objects, I'm
encountering Frank Barknecht's problem with cacheValues and modified
attributes not displaying. This is independent of whether I have a
real object or one of my temp ones. The problem is immediately with
this line of code in _SO_setValue:

        if cls._SO_autoInitDone:
            setattr(self, instanceName(name), value)

because _SO_autoInitDone is never (anywhere in SQLObject.py) set to
true. Now, I can't tell if this is just a missing line somewhere, or
if autoInitDone is an old flag that doesn't have any meaning now that
we're using _SO_selectInit. Ian?

For now, my best-guess solution has been to replace the few instances
of _SO_autoInitDone with _cacheValues where appropriate, to mirror
the setup in addColumn. Seems to be working so far...


2. Is there a good reason SQLObject.destroy(self) was renamed to
SQLObject.destroySelf(self)? It just doesn't read correctly to me,
and breaks some other code we have, and.. I just don't get it. Unless
the intention was to eventually have a cascading delete on the python
side. (we just enforce such things through Postgres, but maybe
there's a case for a having it on the python side.) In which case I'd
still argue to leave .destroy() as the default (since cascade may be
enforced on the db side), and either have an argument 'cascade', or a
separate method .destroyRelated() or similar.

I've changed it back to .destroy() in my copies until I hear
otherwise. :)

Ok, that's all, a little more testing, integrate this into latest CVS
version, and i'll throw it at you.

- Luke

--
 i find your contempt for naked feet curious.

Re: [SQLObject] thoughts on structure and components

[Bud P. B. <bu...@si...>]

On 08 May 2003 09:44:21 -0500
Nick <ni...@dd...> wrote:

> On Thu, 2003-05-08 at 04:37, Bud P.Bruegger wrote:
> > Backend:
> >   An object that manages the interaction with the actual database.
> >   It has the following components:
> >   - mapping
> >   - dbDriver
> >   - dbConnection
> 
> Don't forget cache somewhere, which probably makes more sense to put
> here than in the Connection.

Absolutely right!

> > Mapping:
> >   Description of the physical representation in a dbms-independent way.  
> 
> Isn't mapping dependent on the way the data is stored in the database? 
> Maybe I'm misunderstanding what you mean by mapping.

Yes, the mapping describes how objects are stored in the dbms but
abstracts from the details that differ with the actually used dbms
(mysql, postgres, etc).  

> > Attribute:
> >   A high-level physical type to represent the higher-level physical rep. 
> >   This includes marshalling to and from a lower-level representation. 
> >   Marshalling can do things such as encoding structured objects (phone
> >   number with country and area code) to strings, pickling, mapping a single
> >   attribute (a point) to multiple columns (x, y or lat,long), etc.  
> 
> How does this relate to mapping?  It sounds similar.

It's a component, have a look at the UML class diagram that I
attached..

Thanks for the feedback

--b 

Re: [SQLObject] thoughts on structure and components

[Nick <ni...@dd...>]

On Thu, 2003-05-08 at 04:37, Bud P.Bruegger wrote:
> Backend:
>   An object that manages the interaction with the actual database.
>   It has the following components:
>   - mapping
>   - dbDriver
>   - dbConnection

Don't forget cache somewhere, which probably makes more sense to put
here than in the Connection.

> Mapping:
>   Description of the physical representation in a dbms-independent way.  

Isn't mapping dependent on the way the data is stored in the database? 
Maybe I'm misunderstanding what you mean by mapping.

> Attribute:
>   A high-level physical type to represent the higher-level physical rep.  This
>   includes marshalling to and from a lower-level representation.  Marshalling
>   can do things such as encoding structured objects (phone number with country
>   and area code) to strings, pickling, mapping a single attribute (a point) to
>   multiple columns (x, y or lat,long), etc.  

How does this relate to mapping?  It sounds similar.

Nick

Re: [SQLObject] thoughts on structure and components

[Bud P. B. <bu...@si...>]

Not sure whether the lists does attachments, but here is a UML class
diagram that visualizes my thoughts further

--b

[SQLObject] thoughts on structure and components

[Bud P. B. <bu...@si...>]

Hi everyone,

Been thinking these days and would like to share some of what I came
up with.  I suppose this could be of interest just for academic
purposes or for a possible future refactoring of SQLObject.  If it's
not of interest, just ignore.  

My thinking was triggered by:
- My wish for higher-level physical types that users construct
  depending on the application
- My difficulties to easily add the things to SQLObject that I
  expected to be easy:  e.g., foreign keys...
- I have an intuition (that I cannot materialize) that there is some
  strong coupling in SqlObject that steepens the learning curve and
  makes some behavior unintuitive.  It may be that also the issues of
  difficulties of subclassing etc. are falling in this area...  As I
  said, I have no overly clear ideas to offer.  

So my first product out of this is just a clarification of what
concepts (classes) there should be and how they relate with each
other.  A first rough writeup of my ideas follows here:

Concepts:
---------

Application Object:
  An Application Object is independent of backend storage

Storable:
  A superclass that makes application objects storable.  
  It adds methods that interact with the Backend in order to
  - store
  - update
  - delete
  the object
  In addition, it provided methods to easily access related objects

Backend:
  An object that manages the interaction with the actual database.
  It has the following components:
  - mapping
  - dbDriver
  - dbConnection
  A first version can live with a single driver and connection--a later
  version may be able to work with multiple databases

Connection:
  An actual connection or connection pool that interacts with the dbms

DBdriver:
  A class that provides functionality that depends on the kind of connection
  used, typically depending on:
  - dbms used (e.g., postgres)
  - DBI module (and version)
  The DBdriver supports a subset of dbCols

Mapping:
  Description of the physical representation in a dbms-independent way.  
  The mapping is described by the following components:
  - Attributes of Storable Objects
  - Relationships of Storable Objects

DbCol:  
  A low level physical type to represent a single field in the dbms. It is an
  abstraction of the actual SQL type since different DBdrivers can map it to
  different SQL expressions.  The types are named closely to standard SQL.
  Types such as TIMESTAMP could for example be translated to DateTime by a
  mySQL dbDriver.  

Attribute:
  A high-level physical type to represent the higher-level physical rep.  This
  includes marshalling to and from a lower-level representation.  Marshalling
  can do things such as encoding structured objects (phone number with country
  and area code) to strings, pickling, mapping a single attribute (a point) to
  multiple columns (x, y or lat,long), etc.  

/-----------------------------------------------------------------
| Bud P. Bruegger, Ph.D. 
| Sistema (www.sistema.it)
| Via U. Bassi, 54
| 58100 Grosseto, Italy
| +39-0564-411682 (voice and fax)
\-----------------------------------------------------------------

Re: [SQLObject] Objects with localized properties

[Ian B. <ia...@co...>]

On Wed, 2003-05-07 at 15:20, Luke Opperman wrote:
> This leads me to a vague desire I've had for a while: to be able to do
> selects over joins (really, a filter). Conceptually, something
> equivalent to:
> 
>      def name(self, lang):
>          return [ x for x in self.productNames if x.lang == lang ][0]
> 
> although I'd like a cleaner way to add the condition.
> 
>      def name(self, lang):
>          return self.productNamesWhere(q.lang == lang)[0]
> 
> except this gets into the mess (which i don't entirely mind) of the
> SelectResults system...

I think I understand, and it shouldn't be too hard.  I want to make
joins a bit more powerful anyway (in terms of adding methods to the
class).

Can you give a few more use cases?  That would make it easier for me to
figure out the interface.

  Ian

Re: [SQLObject] Objects with localized properties

[Luke O. <lu...@me...>]

> Yes, that's probably an easier layout too.  In this case a join
> doesn't
> seem quite right... hmmm... you might do:
>
> class Product(SQLObject):
>     _joins = [MultipleJoin('ProductName')]
>
>     def name(self, lang):
>         return ProductName.selectBy(
>             productID=self.id, lang=lang)[0].name

This leads me to a vague desire I've had for a while: to be able to do
selects over joins (really, a filter). Conceptually, something
equivalent to:

     def name(self, lang):
         return [ x for x in self.productNames if x.lang == lang ][0]

although I'd like a cleaner way to add the condition.

     def name(self, lang):
         return self.productNamesWhere(q.lang == lang)[0]

except this gets into the mess (which i don't entirely mind) of the
SelectResults system...

Just some thoughts for down the road. I find my self wanting to filter
joins on a regular basis, and end up doing conditional list
comprehensions like my first example (sql equiv 'where publish <> 0',
etc), but wonder if its worth working it into the core.

- Luke

Re: [SQLObject] Objects with localized properties

[Ian B. <ia...@co...>]

On Wed, 2003-05-07 at 12:38, deelan wrote:
> to clarify as DB schema i was thinking to something
> like this:
> 
> Product table
>    ProductId
>    ...
> 
> ProductName table
>    ProductId
>    Name
>    Lang
> 
> where "lang" is out 2 char lang code, this would require
> a join between the two tables, by doing this i will avoid to
> mess up things when i need to add a name with a new
> localization. if i understood correctly via sqlobject i can
> auto join the two tables. i need to use one of those Join
> objects, is this correct?

Yes, that's probably an easier layout too.  In this case a join doesn't
seem quite right... hmmm... you might do:

class Product(SQLObject):
    _joins = [MultipleJoin('ProductName')]

    def name(self, lang):
        return ProductName.selectBy(
            productID=self.id, lang=lang)[0].name

class ProductName(SQLObject):
    productID = KeyCol(foreignKey='Product')
    name = StringCol()
    lang = StringCol(length=2, varchar=False, notNone=True)


Some might say that ProductName doesn't need a primary key (productID +
lang is unique, and a compound key).  My response is always: then it
should be a join.  Maybe that's true in this case, i.e., some new
LangJoin(...).  I'll think about this.  Creating a specific join of this
type is still mostly theoretical -- I haven't actually done one, and I
don't know if anyone else has either.

  Ian

Re: [SQLObject] Expiring the cache

[Ian B. <ia...@co...>]

On Wed, 2003-05-07 at 10:16, Frank Barknecht wrote:
> Hallo,
> Ian Bicking hat gesagt: // Ian Bicking wrote:
> 
> > Okay... hence the request for a unit test :)  
> 
> I don't know, how do write real unit tests ;(

Look at the tests in tests/test.py, JoinTest2 being the obvious starting
point for your problem (maybe adding another method to it).

Unit tests remove other dependencies and the possibility of the bug
lying elsewhere in the system.  All it comes down to is isolating the
failure case, at which time I expect the problem will be easy to
identify.

Short of a proper unit test, a self-contained script that exemplifies
the problem will do (and is probably easy enough to convert to a unit
test).

  Ian

Re: [SQLObject] Objects with localized properties

[deelan <de...@in...>]

Ian Bicking wrote:

>>probably this solution make more sense:
>>
>>p1 = Product(10)
>>p1.setLocale('it')
>>p1.name <-- get italian name
>>p1.setLocale('en')
>>p1.name <-- get english name
>>
>>finally i would like to avoid doing this:
>>
>>p1 = Product(10)
>>p1.nameEn <-- return english name
>>p1.nameIt <-- return italian name
>>
>>since when a new locale is added i need to add a new variable.
>>what are your thoughts?
> 
> 
> What are you thinking for the database schema?  I imagine there's two
> columns in that case.

to clarify as DB schema i was thinking to something
like this:

Product table
   ProductId
   ...

ProductName table
   ProductId
   Name
   Lang

where "lang" is out 2 char lang code, this would require
a join between the two tables, by doing this i will avoid to
mess up things when i need to add a name with a new
localization. if i understood correctly via sqlobject i can
auto join the two tables. i need to use one of those Join
objects, is this correct?

> 
> One possibilty might be to have a nameEn and nameIt, and do to:
> 
> class Product(SQLObject):
>     ...
>     def _get_name(self):
>         if self.locale == 'en': return self.nameEn
>         elif self.locale == 'it': return self.nameIt
>     def _set_name(self, value):
>         if self.locale == 'en': self.nameEn = value
>         elif self.locale == 'it': self.nameIt = value
> 
> But, assuming locale gets set by the interface somehow, this makes
> Person unthreadsafe for multiple users with multiple languages (since
> typically a single object would be shared among them).  
mmm, that's true. i didn't think to this possibility.


> (...) Then you might get p1.name['en'] -- i.e., name returns a dictionary, that is
> created from name_en, name_it, etc.  Maybe that would be defined like:
> 
> class Product(SQLObject):
>     name = DictOfCols({'it': 'name_it', 'en': 'name_en'})
> 
> But that's just speculative.  Again, without changes you can implement
> _get_name(self): return {'it': self.nameIt, 'en': self.nameEn}, at least
> until I feel clearer implementing something specific.

mmm, this is a nice solution but i'm having hard time to figuring
out the entire thing. i'll think about it.

thanks for the support.

Re: [SQLObject] Expiring the cache

[Frank B. <fb...@fo...>]

Hallo,
Ian Bicking hat gesagt: // Ian Bicking wrote:

> Okay... hence the request for a unit test :)  

I don't know, how do write real unit tests ;(

> I don't think the multi-threaded nature should really cause a
> problem, or at least not this problem, because everything's actually
> happening serially.

Well, the code that still  fails is an application that I have in the
Webware-Sandbox "survey". It's not the current version, but I'll
check that in if I'm back home again...

ciao
-- 
 Frank Barknecht                               _ ______footils.org__

Re: [SQLObject] Expiring the cache

[Ian B. <ia...@co...>]

On Wed, 2003-05-07 at 04:22, Frank Barknecht wrote:
> Ok, major bugs in my scripts: "Change" wasn't called. I now fixed this
> and, well, now it works correct even with joins. Duh. I won't sent the
> fixed version, instead I'll first try my older code to see if I can
> reproduce the bug in the first place. Give me a day...

Okay... hence the request for a unit test :)  I don't think the
multi-threaded nature should really cause a problem, or at least not
this problem, because everything's actually happening serially.

  Ian

Re: [SQLObject] Objects with localized properties

[Ian B. <ia...@co...>]

On Wed, 2003-05-07 at 04:34, deelan wrote:
> p1 = Product(10)
> 
> the object would contain all the possibile names in the implemented 
> locales? what about a:
> 
> p1 = Product(10, 'it')
> p1.name <-- get italian localized name
> 
> to load up just the italian name and description for the product? i 
> think this would case some issues, since if i do:
> 
> p2 = Product(10, 'en')
> 
> p2 is really the same object in the DB but now contains english name and 
> description. probably this solution make more sense:
> 
> p1 = Product(10)
> p1.setLocale('it')
> p1.name <-- get italian name
> p1.setLocale('en')
> p1.name <-- get english name
> 
> finally i would like to avoid doing this:
> 
> p1 = Product(10)
> p1.nameEn <-- return english name
> p1.nameIt <-- return italian name
> 
> since when a new locale is added i need to add a new variable.
> what are your thoughts?

What are you thinking for the database schema?  I imagine there's two
columns in that case.

One possibilty might be to have a nameEn and nameIt, and do to:

class Product(SQLObject):
    ...
    def _get_name(self):
        if self.locale == 'en': return self.nameEn
        elif self.locale == 'it': return self.nameIt
    def _set_name(self, value):
        if self.locale == 'en': self.nameEn = value
        elif self.locale == 'it': self.nameIt = value

But, assuming locale gets set by the interface somehow, this makes
Person unthreadsafe for multiple users with multiple languages (since
typically a single object would be shared among them).  Though if you
turn caching off there's no reason not to share objects among threads
(though you can't do that now).

You could also do p1.name('en'), which is probably marginally better
than p1.nameEn.  You end up programming name() on your own though, which
would be tedious though straightforward.

Essentially it is a compound field (i.e., a single Python field that has
multiple database fields) -- a similar problem perhaps to something like
a point object, where the point is made up of an x and y column.  Then
you might get p1.name['en'] -- i.e., name returns a dictionary, that is
created from name_en, name_it, etc.  Maybe that would be defined like:

class Product(SQLObject):
    name = DictOfCols({'it': 'name_it', 'en': 'name_en'})

But that's just speculative.  Again, without changes you can implement
_get_name(self): return {'it': self.nameIt, 'en': self.nameEn}, at least
until I feel clearer implementing something specific.

Maybe Person(1, 'it') would be possible, at some point, but I'm not sure
how I would implement that now.

  Ian

[SQLObject] Objects with localized properties

[deelan <de...@in...>]

hi there,

i'm new to sqlobject and python in general, i'm developing using .net
framework but i'm dissatisfied of the system for a number of reasons,
one of them is the lack of proper ORM tool. microsoft has objectspace
under development but the current "tech preview" it's just a demo, e.g. 
you can't do many-to-many relations.

so i'm thinking to switch to python, webware, cheetah and sqlobject. 
i've read the docs and i was wondering how can i solve some tipical 
challenges i'm having while i develop web app with .net.

it happens quite often to have to deal with some business objects that 
have multiple strings (names, descriptions, etc.) to help app 
localization. just think about a product in a catalogue that needs two, 
three descriptions based on the locale of the user browsing the page.

how would  you model the object? if you retrieve the object from the
db with:

p1 = Product(10)

the object would contain all the possibile names in the implemented 
locales? what about a:

p1 = Product(10, 'it')
p1.name <-- get italian localized name

to load up just the italian name and description for the product? i 
think this would case some issues, since if i do:

p2 = Product(10, 'en')

p2 is really the same object in the DB but now contains english name and 
description. probably this solution make more sense:

p1 = Product(10)
p1.setLocale('it')
p1.name <-- get italian name
p1.setLocale('en')
p1.name <-- get english name

finally i would like to avoid doing this:

p1 = Product(10)
p1.nameEn <-- return english name
p1.nameIt <-- return italian name

since when a new locale is added i need to add a new variable.
what are your thoughts?

thanks is advance.

Re: [SQLObject] Expiring the cache

[Frank B. <fb...@fo...>]

Hallo,
Frank Barknecht hat gesagt: // Frank Barknecht wrote:

> Attached is my simple Context for Webware. Here the bug happens.

Ok, major bugs in my scripts: "Change" wasn't called. I now fixed this
and, well, now it works correct even with joins. Duh. I won't sent the
fixed version, instead I'll first try my older code to see if I can
reproduce the bug in the first place. Give me a day...

ciao
-- 
 Frank Barknecht                               _ ______footils.org__

Re: [SQLObject] Expiring the cache

[Frank B. <fb...@fo...>]

Hallo,
Ian Bicking hat gesagt: // Ian Bicking wrote:

> Hmm... can you create a unit test that has this bug in it?  I'd added a
> test (JoinTest2) to test.py in which I try to do this, but there's no
> bug happening.

Yes, this test looks okay, but is it running multithreaded or in
Webware?

Attached is my simple Context for Webware. Here the bug happens.
First run JoinCreate.py, then call JoinTest as a Webware page and try
to change some entries.

ciao
-- 
 Frank Barknecht                               _ ______footils.org__

Re: [SQLObject] SQLObject inheritance

[Ian B. <ia...@co...>]

On Tue, 2003-05-06 at 17:54, Ian Bicking wrote:
> I like the idea of subclasses (like Employee) sharing a primary key with
> their superclass, and potentially the superclass having references as to
> which subclass they belong to (from that modeling paper Luke referenced
> a while ago).  But I'm not sure how best to actually implement that --
> both in the database/SQL, and in the Python side.  So that's a future
> goal, and wouldn't replace this current kind of inheritance (just
> provide an alternative).

Hmmm... a thought on the interface for this sort of inheritance:

class Person(SQLObject):

    name = StringCol(length=20)

    class Employee(SubSQLObject):
        salary = CurrencyCol()

Then, would you have Person(1).employee.salary?  Or Person(1).salary
(AttributeError if it's not an employee, or None?).  

This way the superclass knows about the subclasses, as it has to
eventually.  It's more composition than inheritance, but I think that
actually represents the reality of the database better anyway... it's a
funny sort of composition more than it is inheritance (but it can still
be used like inheritance).

As with other declarations, you could do:

class Employee(SubSQLObject):
    ...

Person.addSubSQLObject(Employee)

Whether SubSQLObject would be a placeholder/factory (like Col has
become), or a real class (subclass of SQLObject and MetaSQLObject,
probably), I'm not sure.

Thoughts?

  Ian

[SQLObject] SQLObject inheritance

[Ian B. <ia...@co...>]

I made some changes, and it should now be possible for SQLObject classes
to inherit from each other.  The inheritance structure is not, however,
reflected in the database.  So:

class Person(SQLObject):
    name = StringCol(length=20)

class Employee(Person):
    salary = CurrencyCol()

The employee table is then expected to have both a name and salary
column, and references to Person objects cannot transparently reference
Employee instances (you have to reference one or the other).

I like the idea of subclasses (like Employee) sharing a primary key with
their superclass, and potentially the superclass having references as to
which subclass they belong to (from that modeling paper Luke referenced
a while ago).  But I'm not sure how best to actually implement that --
both in the database/SQL, and in the Python side.  So that's a future
goal, and wouldn't replace this current kind of inheritance (just
provide an alternative).

  Ian

Re: [SQLObject] Expiring the cache

[Ian B. <ia...@co...>]

On Tue, 2003-05-06 at 04:57, Frank Barknecht wrote:
> I did an update but I still get the same wrong behaviour, when I turn
> caching on. To clarify my problem: 
> 
> It only (AFAIK) affects Joins. To give an example using the people.py:
> A Page should make editing the PhoneNumbers possible:

Hmm... can you create a unit test that has this bug in it?  I'd added a
test (JoinTest2) to test.py in which I try to do this, but there's no
bug happening.

There's nothing cached in joins, so I'm not sure why this would be
happening.

  Ian

Re: [SQLObject] Expiring the cache

[Frank B. <fb...@fo...>]

Hallo Ian,
Ian Bicking hat gesagt: // Ian Bicking wrote:

> I think I fixed the caching problem you've been having -- there was some
> old code based on the lazy fetching of columns that was causing a
> problem.  

I did an update but I still get the same wrong behaviour, when I turn
caching on. To clarify my problem: 

It only (AFAIK) affects Joins. To give an example using the people.py:
A Page should make editing the PhoneNumbers possible:

	pid = getPersonIDSomehow()

	p = Person(pid)
	for number in p.phoneNumbers:
		self.writeln("""
		<form>
		<input type="hidden" name="NumberID" value="%d">
		<input type=input name="PhoneNumber" value="%s">
		<input type=input name="Type" value="%s"
		<input type=submit name="_action_Change" value="Change">
		</form>
		""" % (number.id, number.PhoneNumber, number.PhoneType) )

	def Change(self):
		req   = self.request()
		id    = int(req.value('NumberID', None))
		num   = req.value('PhoneNumber', None)
		type  = req.value('Type', None)
		
		p             = PhoneNumber(id)
		p.phoneNumber = num
		p.phoneType   = type

If Change() is called, the database actually gets updated, but if 
the edit form is showed in the response, it doesn't reflect the
changes in the list of PhoneNumbers.

As I said, only Joins like p.phoneNumbers are affected by this. I can,
and always could edit "Person"s directly without the caching errors,
only their Phones are messed up.

ciao
-- 
 Frank Barknecht                               _ ______footils.org__

Re: [SQLObject] Re: problem: implementing BoolCol

[Ian B. <ia...@co...>]

On Mon, 2003-05-05 at 16:57, Nick wrote:
> On Mon, 2003-05-05 at 14:39, Ian Bicking wrote:
> > Validation may be important when fetching from the database, if
> > SQLObject doesn't have total control of the database, and the
> > constraints aren't as restrictive as the validation.  That might be an
> > obscure case, though.
> 
> Conversion is important, especially for DBM and SQLite, which only store
> strings, coming out of the database.  Hoever, *validating* data coming
> back out of the database can be devastating to an application if it
> can't control the data going into the database from another source. 
> You'd never be able to retrieve a row, even to fix the data, if you got
> an exception because the data already in there is bad.

DBMConnection actually is more shelve-like, pickling all its data, so
you get full types in and out.  But if you want to move to another
backend, validation to restrict that flexibility would serve you well,
so it's not beyond all of this.

At first I thought: yes, validating input from the backend isn't very
useful, since there's no good way to recover in the case of invalid
input.

But on second thought, one design decision that I feel is good in
Validator (and its predicessors) is that validation and conversion are a
single process, because they are whether you want them to be or not.

The easiest example would be for an integer (from a string).  The
converter would be int() itself (convenient :).  But then there's an
implicit validation too, because int("abc") isn't going to work.  You
will get an exception.

So you need a way to handle these exceptions, whether or not you want
to.  How you do that isn't clear to me... in some cases they may be an
example of corrupt data in the database (an all too common occurance). 
Throwing an exception is a pain, but really the only thing you can do at
that point is to provide an alternate mechanism so a person can fix
that.  Doing direct queries is one possibility right now.

In another instance, your validation is too tight.  It's a bug, and
should be fixed in your code.  You should always be able to do 
obj.x = obj.x, and I think it deserves an error if something gets in the
way of doing that (and if the validation is really odd or eclectic, you
should be overriding _get_x and _set_x so you can be as eclectic as you
want).  That error will show up eventually -- better sooner than later.

But, in both cases it's a real pain to deal with it automatically or
programmatically.  But it's not clear to me how it should be dealt with.

Using Validator you can actually be explicit about it, like:
  Validator.Int(ifInvalid=None)
So "x" becomes None when you validate it.  Or you could do:
  Validator.Any(Validator.Int(), Validator.Validator())
Which if an input can't be turned into an integer, it returns it as is
(Validator.Validator incidentally being the identity function of
validators). 

That said, I'm still open to more ideas, but handling exceptional
behavior here is a pain in the but, most definitely.  Maybe Validator
could distinguish between validating input and validating output (which
it doesn't currently -- it just distinguishes conversion).  Then you
could at least choose.

I think with some more long-winded programming, even with its features
Validator should be fairly efficient (i.e., multiple implementations of
attemptToPython and attemptFromPython, so you only pay for the features
you use).  Of course, I don't have any performance data at all, so maybe
that's premature optimization...

  Ian

Re: [SQLObject] Re: problem: implementing BoolCol

[Nick <ni...@dd...>]

On Mon, 2003-05-05 at 14:39, Ian Bicking wrote:
> Validation may be important when fetching from the database, if
> SQLObject doesn't have total control of the database, and the
> constraints aren't as restrictive as the validation.  That might be an
> obscure case, though.

Conversion is important, especially for DBM and SQLite, which only store
strings, coming out of the database.  Hoever, *validating* data coming
back out of the database can be devastating to an application if it
can't control the data going into the database from another source. 
You'd never be able to retrieve a row, even to fix the data, if you got
an exception because the data already in there is bad.

Nick