Re: [cx-oracle-users] Garbage collector causes deadlock in multithreaded cx_Oracle server

[Viktor F. <vi...@ch...>]

Hi again.

The code in the server is so large that there would be little meaning in posting it - I spent a day trying to extract just the particular parts of the program that caused the deadlock but all my extracts worked. :-)

However! I wasn't aware of the need to explicitly .close() each cursor - and doing so did the trick! 

The docs just says .close() "closes the cursor now rather than when __del__ is called", which sounds very optional. It didn't occur to me that threaded applications need to close the cursor in any particular order - and I still don't understand exactly why closing a cursor on a .commit():ed connection can cause a deadlock, but that's not important right now.

Once I added an explicit .close() to each cursor, I can no longer force the deadlock!

You have saved my day! Thank you!

Regards,
/Viktor

PS. Since I googled parts of the stack trace (on Linux) to try to find the answer, I figured I'd just post the top of the thread backtrace from gdb in this reply, so that Google may find it if someone else runs into the same problem.

The garbage collector thread:
(gdb) thread apply all bt

Thread 5 (Thread 63114144 (LWP 7621)):
#0  0x00bd67a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x00d575de in __lll_mutex_lock_wait () from /lib/tls/libpthread.so.0
#2  0x00d5420b in _L_mutex_lock_35 () from /lib/tls/libpthread.so.0
#3  0x02fca820 in kpummgl ()
   from /home/viktor/src/instantclient_11_2/libclntsh.so.11.1
#4  0x02ed0274 in ?? ()
   from /home/viktor/src/instantclient_11_2/libclntsh.so.11.1
#5  0xb040f414 in ?? ()
#6  0x09dc9ec0 in ?? ()
#7  0x03c2c870 in ?? ()
#8  0x02bff114 in sltsmna ()
   from /home/viktor/src/instantclient_11_2/libclntsh.so.11.1
#9  0x02bff114 in sltsmna ()
   from /home/viktor/src/instantclient_11_2/libclntsh.so.11.1
#10 0x0126bdae in kpustmtrelease ()
   from /home/viktor/src/instantclient_11_2/libclntsh.so.11.1
#11 0x0124a2eb in OCIStmtRelease ()
   from /home/viktor/src/instantclient_11_2/libclntsh.so.11.1
#12 0x00201720 in Cursor_FreeHandle (self=0x1dc5, raiseException=0)
    at Cursor.c:225
#13 0x0020807d in Cursor_Free (self=0xb7814d88) at Cursor.c:347
#14 0x08089a30 in dict_dealloc (mp=0xb7922714) at Objects/dictobject.c:985
#15 0x080a375a in subtype_dealloc (self=0xb780d68c) at Objects/typeobject.c:999
#16 0x0808981d in PyDict_Clear (op=0xb792224c) at Objects/dictobject.c:891
#17 0x0808bd56 in dict_tp_clear (op=0xb792224c) at Objects/dictobject.c:2088
#18 0x08108445 in collect (generation=1) at Modules/gcmodule.c:769
#19 0x08109125 in _PyObject_GC_Malloc (basicsize=124) at Modules/gcmodule.c:996
#20 0x0810913e in _PyObject_GC_New (tp=0x816faa0) at Modules/gcmodule.c:1467
#21 0x08088a35 in PyDict_New () at Objects/dictobject.c:277
[…]

Other threads:

Thread 4 (Thread 89590688 (LWP 7622)):
#0  0x00bd67a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x00d5789b in __read_nocancel () from /lib/tls/libpthread.so.0
#2  0x02bd4b87 in snttread ()
   from /home/viktor/src/instantclient_11_2/libclntsh.so.11.1
#3  0x02bd3d06 in nttfprd ()
   from /home/viktor/src/instantclient_11_2/libclntsh.so.11.1
#4  0x02bcd3f9 in nsbasic_brc ()
   from /home/viktor/src/instantclient_11_2/libclntsh.so.11.1
#5  0x02bd0466 in nsbrecv ()
   from /home/viktor/src/instantclient_11_2/libclntsh.so.11.1
#6  0x02bc06e1 in nioqrc ()
   from /home/viktor/src/instantclient_11_2/libclntsh.so.11.1
#7  0x02c0b03f in ttcdrv ()
   from /home/viktor/src/instantclient_11_2/libclntsh.so.11.1
#8  0x02bc6548 in nioqwa ()
   from /home/viktor/src/instantclient_11_2/libclntsh.so.11.1
#10 0x02bb6e9c in kpurcsc ()
   from /home/viktor/src/instantclient_11_2/libclntsh.so.11.1
#11 0x02bb1971 in kpuexec ()
   from /home/viktor/src/instantclient_11_2/libclntsh.so.11.1
#12 0x02baf22a in OCIStmtExecute ()
   from /home/viktor/src/instantclient_11_2/libclntsh.so.11.1
#13 0x002087a9 in Cursor_InternalExecute (self=0xb7923770, numIters=Variable "numIters" is not available.
)
    at Cursor.c:563
#14 0x0020ac8d in Cursor_Execute (self=0xb7923770, args=0xb780d16c, 
    keywordArgs=0xb780713c) at Cursor.c:1641
[…]


On 19 jan 2012, at 20:24, Henning von Bargen wrote:

> It would certainly help if you posted code demonstrating the problem.
> 
> Just a wild guess:
> 
> You should always explicitly close() the cursor (not the connection) 
> like this:
> 
> curs = conn.cursor()
> try:
>     cursor.execute(...)
> finally:
>     cursor.close()
> 
> (the with statement can be used in newer Python versions).
> 
> Maybe that helps already.
> 
> The GC might try to close the cursors in a somewhat "wrong" order, which 
> could cause the deadlock.
> On Windows, you might also try to use the ProcessExplorer to see the C 
> function where the thread hangs.
> Is it in an OCI call?
> 
> HTH
> Henning
> 
> ------------------------------------------------------------------------------
> Keep Your Developer Skills Current with LearnDevNow!
> The most comprehensive online learning library for Microsoft developers
> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
> Metro Style Apps, more. Free future releases when you subscribe now!
> http://p.sf.net/sfu/learndevnow-d2d
> _______________________________________________
> cx-oracle-users mailing list
> cx-...@li...
> https://lists.sourceforge.net/lists/listinfo/cx-oracle-users