sqlobject-discuss

[SQLObject] Transaction Implmentation

[Luke O. <lu...@me...>]

Hello all -

(Tried sending this, I guess 70K attachments are just on the edge of
kosher, so get the file here: http://www.amoebagod.com/SQLObject.zip
)

Well, after finding time, and getting today's CVS, I'm very pleased to
announce a test implementation of Python-side Transactions for
SQLObject.

Interface:

New exported class 'Transaction' (yes, I know there's already a
Transaction class in DBConnection, but I couldn't come up with an
alternate name, and they won't clash for now.)

Public Transaction methods:
  t = Transaction()
  t.commit()
  t.rollback()

New methods in SQLObject:

  a = Class.newForTransaction(trans, ...) same as classmethod "new()"
  b = inst.addToTransaction(trans) copies inst to the transaction
(inst remains un-transactioned in this case, although you could do
inst = inst.addToTransaction(trans) )

  inst._persist() is a private func called by Transaction to reset
instance to a usable post-transaction copy.

Example:

from SQLObject import Transaction
from Person import Person

t = Transaction()

a = Person.newForTransaction(t, ...)
b = Person(1).addToTransaction(t)

b.username = 'John'
b.addRole(1)
b.roles
# note: this queries the existing roles for Person(1)
# and mixes in the results.

t.commit()
# note: a and b are now fully functioning regular objects
# that can continue to be used. if possible, b is pulled from
# the cache, and both are now available in the cache as expected.
# t can also be reused at this point, but it has no memory about
# previous efforts. rollback after commit is obviously not supported.

Implementation Notes:

Internally, the transaction is implemented with an in-memory
DBConnection-subclass. This MemoryConnection is not generically
useful, it relies heavily on the previous connections of each of the
objects to provide for data it does not have (especially for joins),
and only maintains a diff from this for joins. Also, many DBAPI
methods are not implemented, as they are not relevant to the limited
accesses used by the new SQLObject methods.

I've tried to be as minimally invasive to SQLObject as feasible, with
only the following changes (beyond adding the three methods mentioned
above):

  __new__() now takes an additional argument 'useCache' to forcefully
avoid caching of a specific instance. This is for addToTransaction,
where we create a clean instance and then switch connections (but
don't want any record in the class/original connection cache)

  new() now sets inst._connection = kw['connection'] if a connection
is passed in. I'm not sure if this will break other expected
behavior, but it seems reasonable to me.

  RelatedJoin: performJoin, add, remove:  these all now use 'inst'
argument instead of self.callingClass to get a connection from.
Again, not sure if this will break expected behavior, but seems
reasonable to me. (I need this because, along with the previous
change, I am expecting true per-object connections when I ask for it.
:)

And that's it. I've attached the fully changed SQLObject directory,
only files with changes are SQLObject.py, Transaction.py (new), and
__init__.py (to import Transaction). Again, this is from CVS an hour
or two ago.

Questions? Comments? Let's go!

- Luke

Re: [SQLObject] Transaction Implmentation

[Ian B. <ia...@co...>]

On Mon, 2003-05-12 at 16:26, Luke Opperman wrote:
> (Tried sending this, I guess 70K attachments are just on the edge of
> kosher, so get the file here: http://www.amoebagod.com/SQLObject.zip
> )

Yeah, and I've forgotten the admin password, so I couldn't pass it
through :-(

I don't fully grok the code yet, but from what I see -- it looks like
most of the changes inside methods in SQLObject are probably the right
way to do things even without these transactions.  

Now, I'm still not sure why there's a distinction between
MemoryConnection and Transaction -- they seem like they could be the
same class.

> Well, after finding time, and getting today's CVS, I'm very pleased to
> announce a test implementation of Python-side Transactions for
> SQLObject.
> 
> Interface:
> 
> New exported class 'Transaction' (yes, I know there's already a
> Transaction class in DBConnection, but I couldn't come up with an
> alternate name, and they won't clash for now.)
> 
> Public Transaction methods:
>   t = Transaction()
>   t.commit()
>   t.rollback()
> 
> New methods in SQLObject:
> 
>   a = Class.newForTransaction(trans, ...) same as classmethod "new()"
>   b = inst.addToTransaction(trans) copies inst to the transaction
> (inst remains un-transactioned in this case, although you could do
> inst = inst.addToTransaction(trans) )

With connections:

trans = MemoryTransaction(persistentConnection)
a = Class.new(connection=trans, ...)
# New copyToConnection method, works between any connections...
b = inst.copyToConnection(trans)

The MemoryTransaction (or whatever it is called -- I feel like "temp"
should be the name somewhere)... it also happens to keep track of things
so it can copy objects back on commit.  It seems like a nice orthogonal
feature, because I can imagine copying between connections besides just
transactions.

Perhaps a different name than "connection" will make this feel
conceptually more clear -- *Connection is something of a misnomer, as
that's only a small part of the functionality, and new things like
DBMConnection and MemoryConnection aren't connections.

>   __new__() now takes an additional argument 'useCache' to forcefully
> avoid caching of a specific instance. This is for addToTransaction,
> where we create a clean instance and then switch connections (but
> don't want any record in the class/original connection cache)

Couldn't the connection's cache just be a sort of no-op, to achieve the
same thing?  That is, MemoryConnection's cache attribute just returns
None on .get(), and ignores .put().  The effect is no caching.

>   new() now sets inst._connection = kw['connection'] if a connection
> is passed in. I'm not sure if this will break other expected
> behavior, but it seems reasonable to me.

Yes, that should be in there anyway.

>   RelatedJoin: performJoin, add, remove:  these all now use 'inst'
> argument instead of self.callingClass to get a connection from.
> Again, not sure if this will break expected behavior, but seems
> reasonable to me. (I need this because, along with the previous
> change, I am expecting true per-object connections when I ask for it.
> :)

Yes, sounds right too.

  Ian

Re: [SQLObject] Transaction Implmentation

[Luke O. <lu...@me...>]

> I don't fully grok the code yet, but from what I see -- it looks
> like
> most of the changes inside methods in SQLObject are probably the
> right
> way to do things even without these transactions.

That's what it seemed like. Cool.

> Now, I'm still not sure why there's a distinction between
> MemoryConnection and Transaction -- they seem like they could be
> the same class.

That was just for my sake as it came together, they could easily be
combined.

> With connections:
>
> trans = MemoryTransaction(persistentConnection)
> a = Class.new(connection=trans, ...)
> # New copyToConnection method, works between any connections...
> b = inst.copyToConnection(trans)

Hmm. yep, that's really all addToTransaction is, and _persist is just
copyBackToClassConnection, which could be the default arg behavior
for copyToConnection.

The one thing I've found handy with the Transaction idea is that it's
an obvious grouping mechanism for a collection of temp objects.
Anyways, the interface you suggest above is very compatible with how
I've built things, so long as we keep
MemoryTransaction.commit()/rollback().

Also, I think it would be good to simplify things somewhat inside
MemoryTransaction to only have a single persistentConnection, as
opposed to tracking per-class connections (which are likely all the
same). Especially since I had to ignore the per-class thing for
joins, where I don't know what class it's coming from. :) Only thing,
I'd probably make it so it'll transparently pick up the connection
from the first object added to it, as opposed to a necessary explicit
part of __init__. (When I'm writing user-side SQLObject code, I never
have direct access to the connection object being used, unless we
want to promote explicit inst._connection access.)


> Perhaps a different name than "connection" will make this feel
> conceptually more clear -- *Connection is something of a misnomer,
> that's only a small part of the functionality, and new things like
> DBMConnection and MemoryConnection aren't connections.

Yes, I like the name MemoryTransaction actually. Although it's not
particularily clearer what it does...

> Couldn't the connection's cache just be a sort of no-op, to achieve
> the same thing?  That is, MemoryConnection's cache attribute just
> returns None on .get(), and ignores .put().

The problem here (why I needed to turn off caching explicitly) was
that copyToConnection first created a new instance using the
old/persistant connection, which we didn't want cached. Now, if we
ensure that MemoryTransaction already has a persistantConnection
(either by setting it explicitly in copyToConnection, or in its
__init__), then this requirement can go away, and behave as you
suggest. Sounds good. I don't care whether MemoryTransaction's cache
picks it up or not, as MemoryTransaction can't currently be used for
a primary connection type anyways.

A side note: I'm having MemoryTransaction use negative object IDS for
newly created temp objects to distinguish and avoid clashes. Anyone
forsee a problem with this? I can't recall seeing a database that
used negative SERIAL/AutoIncrement fields, but I suppose it could
happen...

Let me merge these ideas together, I'll throw up a new email in the
morning. Summary: simplify to single MemoryTransaction class with
single persistantConnection, and single copyToConnection method in
SQLObject. Alrighty.

- Luke