#45 Deadlock Closing Multiple Pools

open
nobody
None
5
2010-08-14
2010-08-14
Anonymous
No

There is an easily reproducible deadlock when closing multiple connection pools concurrently from multiple threads (all versions of C3P0). The problem is that unclosedPooledDataSources.remove(pds) in the registry causes calls to getIdentityToken on other connection pools; this is synchronized and those pools themselves may be blocked on the registry class.monitor. Here are the traces of two blocked threads for 0.9.2-pre1:

Name: GeneralUtil-pool-7-thread-10
State: BLOCKED on java.lang.Class@6282fd0f owned by: GeneralUtil-pool-7-thread-8
Total blocked: 10 Total waited: 11

Stack trace:
com.mchange.v2.c3p0.C3P0Registry.markClosed(C3P0Registry.java:258)
com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.close(AbstractPoolBackedDataSource.java:461)
- locked com.mchange.v2.c3p0.ComboPooledDataSource@57f4ea9d
model.infrastructure.ProcessCopier.stopCircuitBreaker(ProcessCopier.java:113)
- locked model.infrastructure.ProcessCopier@18a61164
model.infrastructure.RASOrchestrator$1.run(RASOrchestrator.java:30)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
java.util.concurrent.FutureTask.run(FutureTask.java:138)
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
java.lang.Thread.run(Thread.java:619)

AND

Name: GeneralUtil-pool-7-thread-8
State: BLOCKED on com.mchange.v2.c3p0.ComboPooledDataSource@57f4ea9d owned by: GeneralUtil-pool-7-thread-10
Total blocked: 9 Total waited: 12

Stack trace:
com.mchange.v2.c3p0.impl.PoolBackedDataSourceBase.getIdentityToken(PoolBackedDataSourceBase.java:80)
com.mchange.v2.c3p0.impl.AbstractIdentityTokenized.equals(AbstractIdentityTokenized.java:40)
java.util.HashMap.removeEntryForKey(HashMap.java:556)
java.util.HashMap.remove(HashMap.java:538)
java.util.HashSet.remove(HashSet.java:216)
com.mchange.v2.c3p0.C3P0Registry.markClosed(C3P0Registry.java:258)
- locked java.lang.Class@6282fd0f
com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.close(AbstractPoolBackedDataSource.java:461)
- locked com.mchange.v2.c3p0.ComboPooledDataSource@57f4ea9d
model.infrastructure.ProcessCopier.stopCircuitBreaker(ProcessCopier.java:113)
- locked model.infrastructure.ProcessCopier@27ce2dd4
model.infrastructure.RASOrchestrator$1.run(RASOrchestrator.java:30)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
java.util.concurrent.FutureTask.run(FutureTask.java:138)
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
java.lang.Thread.run(Thread.java:619)

Discussion