[cx-oracle-users] Garbage collector causes deadlock in multithreaded cx_Oracle server

[Viktor F. <vi...@ch...>]

Hello.

I have a multithreaded RPC server, which spawns a new thread for every incoming RPC request. The backend is an Oracle 11.2 database, and i use cx_Oracle 5.1.1 (all connections started with threaded=True) and Python 2.7 on Linux. I have experienced occasional deadlocks, which have become worse.

I can force a deadlock by having four clients continously call an RPC which performs an UPDATE on the same database row (a semi-realistic scenario). The server is written so that each thread has its own database link, and exactly one cursor on that link. Threads can save links in a global list (protected by a mutex) for later use by other threads. Each thread that reuses an old link creates its own cursor for that link, so cursors are never reused.

I have run gdb on the deadlocked server processes, and can see that one of the threads is always inside a Cursor_Free()-call started by collect() in gcmodule.c. The other threads hang on the Cursor_Execute() running the SQL UPDATE.

If I turn garbage collection off, using "import gc; gc.disable()", the hang does not seem to occur.

It is natural that the threads will compete for the database row lock - once one thread has executed its UPDATE, the other threads' UPDATE:s will (and should) hang until the first thread either commit()s or rollback()s. This behaviour is confirmed by most threads hanging in a Cursor_Execute().

I am pretty certain that no cursors are ever thrown away (and therefore never garbage collected) without having had either .commit() or .rollback() called on their link.

The question therefore becomes: what makes the garbage collection of an old cursor, that once held a database row lock, hang?

I am really at my wits end trying to find what I have done wrong, or what at all is happening.

Does any of this ring a bell with anyone else? I really don't know where to continue looking - my head is already spinning from all the backtraces.

Any thoughts are deeply appreciated.


Thanks in advance,
/Viktor