Thread: [SQLObject] Creating more sqlobjects in _create()
SQLObject is a Python ORM.
Brought to you by:
ianbicking,
phd
From: <ber...@zk...> - 2007-07-05 08:17:41
|
Hello all I'm experiencing frequent deadlocks in my cherrypy/sqlobject multithrea= ded server application. I'm unable to reproduce the error in a small-scale example yet, but one thing I noticed is, that it might be related to me= creating other sqlobjects in an overwritten _create() function. What I see is that various threads try to acquire a threadSafeMethod decorator's lock, originating from various places: ... File "/usr/lib/python2.4/site-packages/sqlobject/sresults.py", line 1= 65, in __iter__ return iter(list(self.lazyIter())) File "/usr/lib/python2.4/site-packages/sqlobject/dbconnection.py", li= ne 804, in next obj =3D self.select.sourceClass.get(result[0], selectResults=3Dresu= lt[1:], connection=3Dself.dbconn) File "/usr/lib/python2.4/site-packages/sqlobject/main.py", line 915, = in get val =3D cls(_SO_fetch_no_create=3D1) File "/usr/lib/python2.4/site-packages/sqlobject/declarative.py", lin= e 92, in _wrapper lock.acquire() File "threading.py", line 100, in acquire rc =3D self.__block.acquire(blocking) And one thread is pending on the cache's lock: ... File "/usr/lib/python2.4/site-packages/sqlobject/declarative.py", lin= e 94, in _wrapper return fn(self, *args, **kwargs) File "/usr/lib/python2.4/site-packages/sqlobject/main.py", line 1214,= in __init__ self._create(id, **kw) File "/usr/lib/python2.4/site-packages/sqlobject/main.py", line 1245,= in _create self._SO_finishCreate(id) File "/usr/lib/python2.4/site-packages/sqlobject/main.py", line 1271,= in _SO_finishCreate cache.created(id, self.__class__, self) File "/usr/lib/python2.4/site-packages/sqlobject/cache.py", line 300,= in created self.caches[cls.__name__].created(id, obj) File "/usr/lib/python2.4/site-packages/sqlobject/cache.py", line 177,= in created self.cull() File "/usr/lib/python2.4/site-packages/sqlobject/cache.py", line 190,= in cull self.lock.acquire() .. originating from a certain _create function (hence my assumption). My create() is looking quite straight-forward, something like this: class MyClass(sqlobject): foo =3D ForeignKey('OtherClass', default=3DNone) bar =3D ForeignKey('OtherClass', default=3DNone) def _create(self, *args, **kw): # Create the object super(MyClass, self)._create(*args, **kw) # Create other object self.foo =3D OtherClass() self.bar =3D OtherClass() So, my question is: is this problematic, and, if yes, what other ways a= re there to do it? Thanks a million Bernhard ... and, sorry for the mess: ___________________________________________________________________ Disclaimer: Diese Mitteilung ist nur fuer die Empfaengerin / den Empfaenger bestimm= t. Fuer den Fall, dass sie von nichtberechtigten Personen empfangen wird, bitten wir diese hoeflich, die Mitteilung an die ZKB zurueckzusenden un= d anschliessend die Mitteilung mit allen Anhaengen sowie allfaellige Kopi= en zu vernichten bzw. zu loeschen. Der Gebrauch der Information ist verbot= en. This message is intended only for the named recipient and may contain confidential or privileged information. If you have received it in error, please advise the sender by return e-= mail and delete this message and any attachments. Any unauthorised use or dissemination of this information is strictly prohibited.= |
From: Oleg B. <ph...@ph...> - 2007-07-05 08:55:44
|
On Thu, Jul 05, 2007 at 10:17:31AM +0200, ber...@zk... wrote: > I'm experiencing frequent deadlocks in my cherrypy/sqlobject multithreaded > server application. I'm unable to reproduce the error in a small-scale > example yet, but one thing I noticed is, that it might be related to me > creating other sqlobjects in an overwritten _create() function. > > What I see is that various threads try to acquire a threadSafeMethod > decorator's lock, originating from various places: Are they different locks? And what are the error messages? Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: <ber...@zk...> - 2007-07-05 09:22:44
|
Hi Oleg, thanks for your response > On Thu, Jul 05, 2007 at 10:17:31AM +0200, ber...@zk... wrot= e: > > I'm experiencing frequent deadlocks in my cherrypy/sqlobject multithreaded > > server application. I'm unable to reproduce the error in a small-sc= ale > > example yet, but one thing I noticed is, that it might be related t= o me > > creating other sqlobjects in an overwritten _create() function. > > > > What I see is that various threads try to acquire a threadSafeMetho= d > > decorator's lock, originating from various places: > > Are they different locks? I don't know, is there an easy way to tell? All I can see at the moment= are the stack traces I have posted. The accesses are made in different threads, to objects of the same clas= s (MyClass in my example). One locks while resolving a ForeignKey in _SO_foreignKey(), one locks in _SO_finishCreate() and the third one in = the selectresult's __iter__(). > And what are the error messages? I have none, the application just locks. Or, more precisely: calls to sqlobject lock, while other parts of the application still work. Bernhard ___________________________________________________________________ Disclaimer: Diese Mitteilung ist nur fuer die Empfaengerin / den Empfaenger bestimm= t. Fuer den Fall, dass sie von nichtberechtigten Personen empfangen wird, bitten wir diese hoeflich, die Mitteilung an die ZKB zurueckzusenden un= d anschliessend die Mitteilung mit allen Anhaengen sowie allfaellige Kopi= en zu vernichten bzw. zu loeschen. Der Gebrauch der Information ist verbot= en. This message is intended only for the named recipient and may contain confidential or privileged information. If you have received it in error, please advise the sender by return e-= mail and delete this message and any attachments. Any unauthorised use or dissemination of this information is strictly prohibited.= |
From: Oleg B. <ph...@ph...> - 2007-07-05 09:32:01
|
On Thu, Jul 05, 2007 at 11:22:36AM +0200, ber...@zk... wrote: > The accesses are made in different threads, to objects of the same class > (MyClass in my example). One locks while resolving a ForeignKey in > _SO_foreignKey(), one locks in _SO_finishCreate() and the third one in the > selectresult's __iter__(). Can you do an experiment - create objects of OtherClass outside of _create() and assign them to foreign keys after the object has been fully created? Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: <ber...@zk...> - 2007-07-05 15:11:39
|
Hi Oleg sql...@li... schrieb am 05.07.2007 11:31:56: > On Thu, Jul 05, 2007 at 11:22:36AM +0200, ber...@zk... wrot= e: > > The accesses are made in different threads, to objects of the same class > > (MyClass in my example). One locks while resolving a ForeignKey in > > _SO_foreignKey(), one locks in _SO_finishCreate() and the third one= in the > > selectresult's __iter__(). > > Can you do an experiment - create objects of OtherClass outside of= > _create() and assign them to foreign keys after the object has been f= ully > created? > I've tried to produce an isolated example of my problem. I'm not yet su= re if everything's needed in here, I'm still trying to buil it further dow= n. Things I noticed so far: - The _create seems not to have anything to do with it, as it locks ev= en without one. - A higher number of threads increase the possibility for deadlocks. - I need to have two classes, MyClass1 and MyClass2, in order for the deadlock to occur. To the code: - On my machine, the code deadlocks after some 10s of objects created.= - The threadframe module may be downloaded here: http://www.majid.info/mylos/stories/2004/06/10/threadframe.html. Maybe = you have some clever debugger that can do that for you... :-) Thanks! Bernhard #**************************************************** import sqlobject import threading import time sqlobject.sqlhub.processConnection =3D sqlobject.connectionForURI('mysql://XXX') class Attribute(sqlobject.SQLObject): """ Attribute class """ attributeSet =3D sqlobject.ForeignKey('AttributeSet') name =3D sqlobject.StringCol(default =3D '') value =3D sqlobject.StringCol(default =3D '') class AttributeSet(sqlobject.SQLObject): """ Attribute set class, contains attributes """ attributes =3D sqlobject.SQLMultipleJoin('Attribute') def _set_attributes(self, attrs): # First clear... for attr in self.attributes: attr.destroySelf() # ... then set. for k, v in attrs.items(): Attribute(attributeSet =3D self, name =3D k, value =3D v) def _get_attributes(self): attrs =3D {} for attribute in list(self._SO_get_attributes()): attrs[attribute.name] =3D attribute.value return attrs class MyClass1(sqlobject.SQLObject): """ First class that uses an attribute set. """ attrSet =3D sqlobject.ForeignKey('AttributeSet', default =3D None) # I would want to create the attribute set here, but for # Oleg's suggestion, I'm doing it from the calling class. #def _create(self, *args, **kw): #super(MyClass1, self)._create(*args, **kw) #self.attrSet =3D AttributeSet() #self.attrSet.attributes =3D dict(foo =3D 1, bar =3D 2) class MyClass2(sqlobject.SQLObject): """ Second class that uses an attribute set. """ attrSet =3D sqlobject.ForeignKey('AttributeSet', default =3D None) #def _create(self, *args, **kw): #super(MyClass2, self)._create(*args, **kw) #self.attrSet =3D AttributeSet() #self.attrSet.attributes =3D dict(foo =3D 1, bar =3D 2) # Create tables. Attribute.createTable(ifNotExists =3D True) AttributeSet.createTable(ifNotExists =3D True) MyClass1.createTable(ifNotExists =3D True) MyClass2.createTable(ifNotExists =3D True) class MyThread(threading.Thread): """ A thread that produces instances of MyClass1 or Myclass2. """ def __init__(self, Klass): super(MyThread, self).__init__() self._Klass =3D Klass def run(self): print '%02d %02d %20s : Started' % (time.localtime().tm_min, time.localtime().tm_sec, threading.currentThread().getName()) for i in range(100): mc =3D self._Klass() mc.attrSet =3D AttributeSet() mc.attrSet.attributes =3D dict(foo =3D 1, bar =3D 2) # Accessing these will result in a deadlock for key, value in mc.attrSet.attributes.items(): pass if i % 10 =3D=3D 0: print '%02d %02d %20s : %s' % (time.localtime().tm_min,= time.localtime().tm_sec, threading.currentThread().getName(), i) # Create some threads threads =3D [MyThread(MyClass1) for i in range(10)] + [MyThread(MyClass= 2) for i in range(10)] print 'Starting threads' for t in threads: t.start() time.sleep(10) import threadframe, traceback, sys frames =3D threadframe.threadframe() print '-' * 72 for frame in frames: print '-' * 72 print 'frame ref count =3D %d' % sys.getrefcount(frame) traceback.print_stack(frame, file=3Dsys.stdout) for t in threads: t.join() print 'Done' #**************************************************** ___________________________________________________________________ Disclaimer: Diese Mitteilung ist nur fuer die Empfaengerin / den Empfaenger bestimm= t. Fuer den Fall, dass sie von nichtberechtigten Personen empfangen wird, bitten wir diese hoeflich, die Mitteilung an die ZKB zurueckzusenden un= d anschliessend die Mitteilung mit allen Anhaengen sowie allfaellige Kopi= en zu vernichten bzw. zu loeschen. Der Gebrauch der Information ist verbot= en. This message is intended only for the named recipient and may contain confidential or privileged information. If you have received it in error, please advise the sender by return e-= mail and delete this message and any attachments. Any unauthorised use or dissemination of this information is strictly prohibited.= |
From: <ber...@zk...> - 2007-07-06 12:46:46
|
Hi all I think I found the cause of my deadlock, which is (if I'm right that i= s), located in main.py:912: val =3D cache.get(id, cls) if val is None: try: val =3D cls(_SO_fetch_no_create=3D1) We reach this place e.g. by resolving a foreign key through _SO_foreignKey() and get(). If I read the code correctly, the cache's lock is acquired in cache.get= () (but not released, which would be done in cache.finishPut(cls)), if the= object doesn't already exist. cls(_SO_fetch_no_create=3D1) then acquire= s the class' lock through the threadSafeMethod() wrapper in declarative.py. On the other hand, if I'm creating an object of the same type, at exact= ly the time between the cache.get() and cls() calls, I'll be able to acqui= re the class' lock and then pend on the cache's lock, through threadSafe __init__(), then _create(), then _SO_finishCreate(), cacheSet.created()= , CacheFactory.created() and finally CashFactory.cull(). This leaves us with the first thread having acquired the cache's lock w= hile pending for the class' lock, and the second thread having acquired the class' lock and pending for the cache's lock -> deadlock. The situation= can only occur, if the creation of the object triggers a cache.cull(). In a= ny other case, the cache isn't locked and thus there are no problems. Does this all make sense? Bernhard sql...@li... schrieb am 05.07.2007 17:11:20: > Hi Oleg > > sql...@li... schrieb am 05.07. > 2007 11:31:56: > > > On Thu, Jul 05, 2007 at 11:22:36AM +0200, ber...@zk... wr= ote: > > > The accesses are made in different threads, to objects of the sam= e class > > > (MyClass in my example). One locks while resolving a ForeignKey i= n > > > _SO_foreignKey(), one locks in _SO_finishCreate() and the third o= ne in the > > > selectresult's __iter__(). > > > > Can you do an experiment - create objects of OtherClass outside = of > > _create() and assign them to foreign keys after the object has been= fully > > created? > > > > I've tried to produce an isolated example of my problem. I'm not yet > sure if everything's needed in here, I'm still trying to buil it furt= her down. > > Things I noticed so far: > > - The _create seems not to have anything to do with it, as it locks > even without one. > - A higher number of threads increase the possibility for deadlocks.= > - I need to have two classes, MyClass1 and MyClass2, in order for > the deadlock to occur. > > To the code: > - On my machine, the code deadlocks after some 10s of objects create= d. > - The threadframe module may be downloaded here: http://www.majid. > info/mylos/stories/2004/06/10/threadframe.html. Maybe you have some > clever debugger that can do that for you... :-) > ___________________________________________________________________ Disclaimer: Diese Mitteilung ist nur fuer die Empfaengerin / den Empfaenger bestimm= t. Fuer den Fall, dass sie von nichtberechtigten Personen empfangen wird, bitten wir diese hoeflich, die Mitteilung an die ZKB zurueckzusenden un= d anschliessend die Mitteilung mit allen Anhaengen sowie allfaellige Kopi= en zu vernichten bzw. zu loeschen. Der Gebrauch der Information ist verbot= en. This message is intended only for the named recipient and may contain confidential or privileged information. If you have received it in error, please advise the sender by return e-= mail and delete this message and any attachments. Any unauthorised use or dissemination of this information is strictly prohibited.= |
From: Oleg B. <ph...@ph...> - 2007-07-06 13:01:12
|
On Fri, Jul 06, 2007 at 02:46:38PM +0200, ber...@zk... wrote: > I think I found the cause of my deadlock, which is (if I'm right that is), > located in main.py:912: > > val = cache.get(id, cls) > if val is None: > try: > val = cls(_SO_fetch_no_create=1) > > We reach this place e.g. by resolving a foreign key through > _SO_foreignKey() and get(). > > If I read the code correctly, the cache's lock is acquired in cache.get() > (but not released, which would be done in cache.finishPut(cls)), if the > object doesn't already exist. cls(_SO_fetch_no_create=1) then acquires the > class' lock through the threadSafeMethod() wrapper in declarative.py. > > On the other hand, if I'm creating an object of the same type, at exactly > the time between the cache.get() and cls() calls, I'll be able to acquire > the class' lock and then pend on the cache's lock, through threadSafe > __init__(), then _create(), then _SO_finishCreate(), cacheSet.created(), > CacheFactory.created() and finally CashFactory.cull(). > > This leaves us with the first thread having acquired the cache's lock while > pending for the class' lock, and the second thread having acquired the > class' lock and pending for the cache's lock -> deadlock. The situation can > only occur, if the creation of the object triggers a cache.cull(). In any > other case, the cache isn't locked and thus there are no problems. > > Does this all make sense? Absolutely. Thank you for the excellent analysis! But could the cache release the lock before the new row (SQLObject) is put to the cache? I doubt it. Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: <ber...@zk...> - 2007-07-06 13:50:30
|
> On Fri, Jul 06, 2007 at 02:46:38PM +0200, ber...@zk... wrot= e: > > I think I found the cause of my deadlock, which is (if I'm right th= at is), > > located in main.py:912: > > > > val =3D cache.get(id, cls) > > if val is None: > > try: > > val =3D cls(_SO_fetch_no_create=3D1) > > > > We reach this place e.g. by resolving a foreign key through > > _SO_foreignKey() and get(). > > > > If I read the code correctly, the cache's lock is acquired in cache.get() > > (but not released, which would be done in cache.finishPut(cls)), if= the > > object doesn't already exist. cls(_SO_fetch_no_create=3D1) then acq= uires the > > class' lock through the threadSafeMethod() wrapper in declarative.p= y. > > > > On the other hand, if I'm creating an object of the same type, at exactly > > the time between the cache.get() and cls() calls, I'll be able to acquire > > the class' lock and then pend on the cache's lock, through threadSa= fe > > __init__(), then _create(), then _SO_finishCreate(), cacheSet.created(), > > CacheFactory.created() and finally CashFactory.cull(). > > > > This leaves us with the first thread having acquired the cache's lo= ck while > > pending for the class' lock, and the second thread having acquired = the > > class' lock and pending for the cache's lock -> deadlock. The situa= tion can > > only occur, if the creation of the object triggers a cache.cull(). = In any > > other case, the cache isn't locked and thus there are no problems. > > > > Does this all make sense? > > Absolutely. Thank you for the excellent analysis! But could the ca= che > release the lock before the new row (SQLObject) is put to the cache? = I > doubt it. I'm not (yet) an expert on sqlobject's inner goings, but I'd say the sa= me. Otherwise it would be possible for some other thread to start fetching = the same object again. My hacky solution is to not acquire the class's lock in __init__(), whe= n in the process of fetching an instance: def threadSafeMethod(lock): def decorator(fn): def _wrapper(self, *args, **kwargs): if '_SO_fetch_no_create' in kwargs: return fn(self, *args, **kwargs) [...] If I see that correctly, the class lock isn't really needed in __init__= () while fetching. But there should certainly be nicer way to do that. Bernhard ___________________________________________________________________ Disclaimer: Diese Mitteilung ist nur fuer die Empfaengerin / den Empfaenger bestimm= t. Fuer den Fall, dass sie von nichtberechtigten Personen empfangen wird, bitten wir diese hoeflich, die Mitteilung an die ZKB zurueckzusenden un= d anschliessend die Mitteilung mit allen Anhaengen sowie allfaellige Kopi= en zu vernichten bzw. zu loeschen. Der Gebrauch der Information ist verbot= en. This message is intended only for the named recipient and may contain confidential or privileged information. If you have received it in error, please advise the sender by return e-= mail and delete this message and any attachments. Any unauthorised use or dissemination of this information is strictly prohibited.= |
From: Oleg B. <ph...@ph...> - 2007-07-12 15:00:28
|
On Fri, Jul 06, 2007 at 03:50:24PM +0200, ber...@zk... wrote: > My hacky solution is to not acquire the class's lock in __init__(), when in > the process of fetching an instance: > > def threadSafeMethod(lock): > def decorator(fn): > def _wrapper(self, *args, **kwargs): > > if '_SO_fetch_no_create' in kwargs: > return fn(self, *args, **kwargs) Does the solution work in the long run? Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: <ber...@zk...> - 2007-07-16 13:41:28
|
Hi Oleg > On Fri, Jul 06, 2007 at 03:50:24PM +0200, ber...@zk... wrot= e: > > My hacky solution is to not acquire the class's lock in __init__(),= when in > > the process of fetching an instance: > > > > def threadSafeMethod(lock): > > def decorator(fn): > > def _wrapper(self, *args, **kwargs): > > > > if '_SO_fetch_no_create' in kwargs: > > return fn(self, *args, **kwargs) > > Does the solution work in the long run? Seems like it. I haven't had a single lockup since then, and no negativ= e side-effects either. Bernhard ___________________________________________________________________ Disclaimer: Diese Mitteilung ist nur fuer die Empfaengerin / den Empfaenger bestimm= t. Fuer den Fall, dass sie von nichtberechtigten Personen empfangen wird, bitten wir diese hoeflich, die Mitteilung an die ZKB zurueckzusenden un= d anschliessend die Mitteilung mit allen Anhaengen sowie allfaellige Kopi= en zu vernichten bzw. zu loeschen. Der Gebrauch der Information ist verbot= en. This message is intended only for the named recipient and may contain confidential or privileged information. If you have received it in error, please advise the sender by return e-= mail and delete this message and any attachments. Any unauthorised use or dissemination of this information is strictly prohibited.= |
From: Oleg B. <ph...@ph...> - 2007-07-17 14:15:41
|
On Mon, Jul 16, 2007 at 03:41:19PM +0200, ber...@zk... wrote: > > > def threadSafeMethod(lock): > > > def decorator(fn): > > > def _wrapper(self, *args, **kwargs): > > > > > > if '_SO_fetch_no_create' in kwargs: > > > return fn(self, *args, **kwargs) > > > > Does the solution work in the long run? > > Seems like it. I haven't had a single lockup since then, and no negative > side-effects either. Applied and committed in the revisions 2758-2762 (branches 0.7, 0.8, 0.9, the trunk and the docs). Thank you! Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |