Re: [C3p0-users] C3P0 Deadlock Issues

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Peter,

No problem on the direct e-mail, especially with a detailed report  
and specific concern.

c3p0 has problems with scenarios where database operations neither  
succeed nor fail promptly with an Exception. c3p0-0.9.1-pre7 and  
above (-pre10 will be released this evening, so you might wait 'til  
tomorrow to upgrade) include a configuration parameter called  
maxAdministrativeTaskTime that you can use to force "hung" operations  
to fail promptly, working around this issue. Try setting this (5  
seconds is probably reasonable), and see if the behavior doesn't  
improve. (It almost certainly will, if the database acquisition  
attempts hitting the locked DB are well-behaved and respond to an  
interrupt().)

Note that maxAdministrativeTaskTime is not settable in hibernate's  
config file. You'll have to create a file called c3p0.properties in  
the top-level of your classpath (or c3p0-config.xml for more advanced  
configuration options). Please see c3p0's docs for more. Anyway, the  
properties file might be as simple as...

c3p0.maxAdministrativeTaskTime=3

...though there are lots of other exciting config params you can add  
if you want. Note that the 6 params you include must be set via  
hibernate's config (as hibernate overwrites the c3p0.properties  
settings for these params), while all other params must be set in  
c3p0.properties. (I've corresponded with the hibernate guys in hopes  
of making this less confusing in the future, but this is where we are  
now.)

Other things you could try are, don't set "timeout" (or set it to 0)  
and don't set idle_test_period (or set it to 0). Then, if there are  
no client requests while your database is frozen, you might get lucky  
and c3p0 might not try to acquire Connections during your down time.  
But this is fragile, you probably do want Connection expiration and  
testing, and there may well still be clients.  
maxAdministrativeTaskTime is your best bet overall. I'd leave your  
settings alone, and go with that.

If you want to understand what's going on:

1) c3p0 maintains a fixed size pool of threads, which is responsible  
for Connection acquisition, destruction, and testing. (This is the  
secret to c3p0's high concurrency -- all slow DB operations are done  
asynchronously and outside the pool's lock.

2) c3p0 decides it needs to acquire more Connections while your  
database is locked. (It does so because it has timed out Connections  
per your config, and dropped below minPoolSize, because Connection  
tests have failed, dropping the pool below minPoolSize, or because of  
client load as  all Connections are checked-out by (hung) clients  
while your database does its thing.

3) c3p0 sends Connection acquisition tasks to its Thread pool. Those  
tasks don't succeed, don't fail, but hang for a long period of time.  
All Threads in the pool become occupied by hung tasks.

4) After some time, the Thread pool decides it is deadlocked. It  
replaces the occupied threads with fresh threads, and creates single- 
use threads to manage its backlog of unattended tasks. All these  
threads get hung too, since database acquisition tasks neither  
succeed nor fail.

5) Eventually, c3p0 interrupt()s the hung threads to try to prevent a  
memory/Thread leak. But these bad events happen over and over again.

Theoretically c3p0 should recover when your database comes back on  
line. I'm not sure why it doesn't for you. I'll try to reproduce your  
scenario, and see if I can't improve c3p0's default behavior.

Anyway, maxAdministrativeTaskTime prevents things from getting out of  
hand by forcing (hopefully) the hung threads to throw an Exception  
rather than freeze, preventing c3p0 from having to spawn all kinds of  
new Threads to work around the apparent deadlock. Please give it a try.

      smiles,
          Steve

---
Steve Waldman
Machinery For Change, Inc.

On Oct 23, 2006, at 4:07 PM, Pet...@md... wrote:

>
> Steve,
>
> Apologies for the direct nature of this email - I am using C3P0 in  
> a system testing environment and am experiencing deadlock issues. I  
> was wondering whether you could shed some light on any possible  
> resolutions, or at the very least find this information useful for  
> future releases.
>
> I am running a J2EE Application, which is run from an IBM Websphere  
> 5.0 App Server. The application uses JBoss jBPM, which under the  
> covers uses Hibernate for persistence and is configured to use  
> c3p0. We are currently running c3p0 version 0.9.0 and we are using  
> DB2/400 as our database server.
>
> The issue which we regularly come up against is, almost daily there  
> is a scheduled backup occurring on the database server. As part of  
> this, the backup process acquires exclusive locks on all tables and  
> these are not released until the backup is complete. This makes it  
> impossible to communicate with the database during this period. As  
> a result, during the period attempting to acquire a SQL Connection  
> results in an Exception being thrown.
>
> However, we demonstrated a fairly simple piece of code, which  
> contains two threads - a producer and a consumer that run in a  
> loop. This loop contains a catch (HibernateException he) block,  
> which logs any Hibernate exceptions but continues with processing.  
> This code demonstrated that during the backup, it simply logs the  
> Exceptions but keeps on retrying, when the backup is complete the  
> application continues as normal. We demonstrated this application  
> using Hibernate on it's own, with C3P0 as the pool and with DBCP.
>
> However, when we run our application in the same environment, we  
> come up against serious problems. First of all, our settings are:
>
> hibernate.c3p0.min_size=1
> hibernate.c3p0.max_size=50
> hibernate.c3p0.timeout=5000
> hibernate.c3p0.max_statements=0
> hibernate.c3p0.idle_test_period=300
> hibernate.c3p0.acquire_increment=2
>
> I've attached the logs at the bottom of this email, however to save  
> you from trawling through them - I've included below the main events:
>
> This illustrates the first log message since the backup starting  
> and in turn acquiring the exclusive lock on the database.
>
> 02:08:27,492 [EngineThread] ERROR waitForQueueItemAvailable 
> (EngineThread : Error whilst retrieving next Queue Item
> com.mdsuk.bpm.engine.queue.QueueException:  
> org.hibernate.exception.GenericJDBCException: Cannot open connection
>         at  
> com.mdsuk.bpm.engine.queue.impl.QueueDAOImpl.getQueueItemForExecution( 
> QueueDAOImpl.java(Compiled Code))
>         at com.mdsuk.bpm.engine.queue.Queue.getNextQueueItem 
> (Queue.java(Compiled Code))
>         at  
> com.mdsuk.bpm.engine.core.EngineThread.waitForQueueItemAvailable 
> (EngineThread.java(Compiled Code))
>         at com.mdsuk.bpm.engine.core.EngineThread.getNextQueueItem 
> (EngineThread.java:242)
>         at com.mdsuk.bpm.engine.core.EngineThread.run 
> (EngineThread.java:131)
>         at java.lang.Thread.run(Thread.java:568)
> Caused by: org.hibernate.exception.GenericJDBCException: Cannot  
> open connection
>         at  
> org.hibernate.exception.SQLStateConverter.handledNonSpecificException( 
> SQLStateConverter.java(Inlined Compiled Code))
>         at org.hibernate.exception.SQLStateConverter.convert 
> (SQLStateConverter.java(Compiled Code))
>         at org.hibernate.exception.JDBCExceptionHelper.convert 
> (JDBCExceptionHelper.java(Compiled Code))
>         at org.hibernate.exception.JDBCExceptionHelper.convert 
> (JDBCExceptionHelper.java(Inlined Compiled Code))
>         at org.hibernate.jdbc.ConnectionManager.openConnection 
> (ConnectionManager.java(Compiled Code))
>         at org.hibernate.jdbc.ConnectionManager.getConnection 
> (ConnectionManager.java(Inlined Compiled Code))
>         at org.hibernate.jdbc.JDBCContext.connection 
> (JDBCContext.java(Compiled Code))
>         at org.hibernate.transaction.JDBCTransaction.begin 
> (JDBCTransaction.java(Compiled Code))
>         at org.hibernate.impl.SessionImpl.beginTransaction 
> (SessionImpl.java(Compiled Code))
>         at org.jbpm.persistence.db.DbPersistenceService.getSession 
> (DbPersistenceService.java(Compiled Code))
>         at org.jbpm.JbpmContext.getSession(JbpmContext.java 
> (Compiled Code))
>         ... 6 more
> Caused by: java.sql.SQLException: Connections could not be acquired  
> from the underlying database!
>         at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java 
> (Compiled Code))
>         at  
> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnec 
> tion(C3P0PooledConnectionPool.java(Compiled Code))
>         at com.mchange.v2.c3p0.PoolBackedDataSource.getConnection 
> (PoolBackedDataSource.java(Compiled Code))
>         at  
> org.hibernate.connection.C3P0ConnectionProvider.getConnection 
> (C3P0ConnectionProvider.java(Compiled Code))
>         ... 13 more
> Caused by:  
> com.mchange.v2.resourcepool.CannotAcquireResourceException: A  
> ResourcePool could not acquire a resource from its primary factory  
> or source.
>         at  
> com.mchange.v2.resourcepool.BasicResourcePool.awaitAcquire 
> (BasicResourcePool.java(Compiled Code))
>         at  
> com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource 
> (BasicResourcePool.java(Compiled Code))
>         ... 16 more
> 02:08:39,359 [com.mchange.v2.async.ThreadPoolAsynchronousRunner 
> $PoolThread-#0] WARN  run(BasicResourcePool :  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@2ddeaf97  
> -- Acquisition Attempt Failed!!! Clearing pending acquires. While  
> trying to acquire a needed new resource, we failed to succeed more  
> than the maximum number of allowed acquisition attempts (30).
>
> [21/10/06 02:08:39:360 BST] 269bafb0 JDBCException W  
> org.hibernate.util.JDBCExceptionReporter  SQL Error: 0, SQLState: null
>
> This carries on for some time, later on the following message is  
> logged:
>
> 02:41:18,835 [Thread-12] WARN  run(ThreadPoolAsynchronousRunner :  
> com.mchange.v2.async.ThreadPoolAsynchronousRunner 
> $DeadlockDetector@64b2f9b -- APPARENT DEADLOCK!!! Creating  
> emergency threads for unassigned pending tasks!
> 02:41:18,835 [Thread-12] WARN  run(ThreadPoolAsynchronousRunner :  
> com.mchange.v2.async.ThreadPoolAsynchronousRunner 
> $DeadlockDetector@64b2f9b -- APPARENT DEADLOCK!!! Complete Status:  
> [num_managed_threads: 3, num_active: 3; activeTasks:  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@f252f8f  
> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1),  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@38c72f82  
> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2),  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@daaef99  
> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0);  
> pendingTasks: com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@7247af99, com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@708f2f9e, com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@1dd7af95, com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@6d366f8b, com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@2cce6f8f, com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@5c74efb5, com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@b79efa6, com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@872f9e, com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@567daf97, com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@4818af97, com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@6588ef8b, com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@3ff32f8d, com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@67522f82, com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@2b15efb5, com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@1de72f99, com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@6db2af9e, com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@df52f97, com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@6e89ef8b, com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@5d7a6f8f, com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@75ce2fb4, com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@18616f99, com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@3411ef9f, com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@54192f97, com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@796fef89, com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@60acef8f, com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@6a392f8e, com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@ddf2fb4, com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@4c4aef98, com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@59c6af9e, com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@7d86ef97, com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@67a9af8b, com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@7ffc2f8c]
> 02:41:28,443 [com.mchange.v2.async.ThreadPoolAsynchronousRunner 
> $PoolThread-#1] WARN  run(BasicResourcePool :  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@f252f8f  
> -- Acquisition Attempt Failed!!! Clearing pending acquires. While  
> trying to acquire a needed new resource, we failed to succeed more  
> than the maximum number of allowed acquisition attempts (30).
>
> Followed by a few more [EngineThread] ERROR  
> waitForQueueItemAvailable( messages, then followed by:
>
> 02:42:16,952 [Task-Thread-for- 
> com.mchange.v2.async.ThreadPerTaskAsynchronousRunner@39202f81]  
> WARN  run(BasicResourcePool :  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@708f2f9e  
> -- Acquisition Attempt Failed!!! Clearing pending acquires. While  
> trying to acquire a needed new resource, we failed to succeed more  
> than the maximum number of allowed acquisition attempts (30).
> 02:42:17,097 [Task-Thread-for- 
> com.mchange.v2.async.ThreadPerTaskAsynchronousRunner@39202f81]  
> WARN  run(BasicResourcePool :  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1dd7af95  
> -- Acquisition Attempt Failed!!! Clearing pending acquires. While  
> trying to acquire a needed new resource, we failed to succeed more  
> than the maximum number of allowed acquisition attempts (30).
> 02:42:17,237 [Task-Thread-for- 
> com.mchange.v2.async.ThreadPerTaskAsynchronousRunner@39202f81]  
> WARN  run(BasicResourcePool :  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@6d366f8b  
> -- Acquisition Attempt Failed!!! Clearing pending acquires. While  
> trying to acquire a needed new resource, we failed to succeed more  
> than the maximum number of allowed acquisition attempts (30).
> 02:42:17,376 [Task-Thread-for- 
> com.mchange.v2.async.ThreadPerTaskAsynchronousRunner@39202f81]  
> WARN  run(BasicResourcePool :  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@2cce6f8f  
> -- Acquisition Attempt Failed!!! Clearing pending acquires. While  
> trying to acquire a needed new resource, we failed to succeed more  
> than the maximum number of allowed acquisition attempts (30).
> 02:42:17,518 [Task-Thread-for- 
> com.mchange.v2.async.ThreadPerTaskAsynchronousRunner@39202f81]  
> WARN  run(BasicResourcePool :  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@5c74efb5  
> -- Acquisition Attempt Failed!!! Clearing pending acquires. While  
> trying to acquire a needed new resource, we failed to succeed more  
> than the maximum number of allowed acquisition attempts (30).
> 02:42:17,661 [Task-Thread-for- 
> com.mchange.v2.async.ThreadPerTaskAsynchronousRunner@39202f81]  
> WARN  run(BasicResourcePool :  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@872f9e --  
> Acquisition Attempt Failed!!! Clearing pending acquires. While  
> trying to acquire a needed new resource, we failed to succeed more  
> than the maximum number of allowed acquisition attempts (30).
> 02:42:17,946 [Task-Thread-for- 
> com.mchange.v2.async.ThreadPerTaskAsynchronousRunner@39202f81]  
> WARN  run(BasicResourcePool :  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@b79efa6  
> -- Acquisition Attempt Failed!!! Clearing pending acquires. While  
> trying to acquire a needed new resource, we failed to succeed more  
> than the maximum number of allowed acquisition attempts (30).
> 02:42:18,243 [Task-Thread-for- 
> com.mchange.v2.async.ThreadPerTaskAsynchronousRunner@39202f81]  
> WARN  run(BasicResourcePool :  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@567daf97  
> -- Acquisition Attempt Failed!!! Clearing pending acquires. While  
> trying to acquire a needed new resource, we failed to succeed more  
> than the maximum number of allowed acquisition attempts (30).
> 02:42:18,539 [Task-Thread-for- 
> com.mchange.v2.async.ThreadPerTaskAsynchronousRunner@39202f81]  
> WARN  run(BasicResourcePool :  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@4818af97  
> -- Acquisition Attempt Failed!!! Clearing pending acquires. While  
> trying to acquire a needed new resource, we failed to succeed more  
> than the maximum number of allowed acquisition attempts (30).
> 02:42:18,905 [Thread-12] WARN  run(ThreadPoolAsynchronousRunner :  
> com.mchange.v2.async.ThreadPoolAsynchronousRunner 
> $DeadlockDetector@64b2f9b -- APPARENT DEADLOCK!!! Creating  
> emergency threads for unassigned pending tasks!
> 02:42:18,905 [Thread-12] WARN  run(ThreadPoolAsynchronousRunner :  
> com.mchange.v2.async.ThreadPoolAsynchronousRunner 
> $DeadlockDetector@64b2f9b -- APPARENT DEADLOCK!!! Complete Status:  
> [num_managed_threads: 3, num_active: 3; activeTasks:  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@55cfaf8e  
> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1),  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1fe9af91  
> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0),  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@d4af92  
> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2);  
> pendingTasks: com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@8d92f95]
> 02:42:31,822 [com.mchange.v2.async.ThreadPoolAsynchronousRunner 
> $PoolThread-#0] WARN  run(BasicResourcePool :  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1fe9af91  
> -- Acquisition Attempt Failed!!! Clearing pending acquires. While  
> trying to acquire a needed new resource, we failed to succeed more  
> than the maximum number of allowed acquisition attempts (30).
>
> Up to this point, the logs are for every few seconds, however this  
> changes to:
>
> 02:44:09,015 [Thread-12] WARN  run(ThreadPoolAsynchronousRunner :  
> com.mchange.v2.async.ThreadPoolAsynchronousRunner 
> $DeadlockDetector@64b2f9b -- APPARENT DEADLOCK!!! Creating  
> emergency threads for unassigned pending tasks!
> 02:44:09,015 [Thread-12] WARN  run(ThreadPoolAsynchronousRunner :  
> com.mchange.v2.async.ThreadPoolAsynchronousRunner 
> $DeadlockDetector@64b2f9b -- APPARENT DEADLOCK!!! Complete Status:  
> [num_managed_threads: 3, num_active: 3; activeTasks:  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@4f8fef92  
> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2),  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@7d792f8e  
> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0),  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@5106f9d  
> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1);  
> pendingTasks: com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@7e71ef81, com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@7a83af92]
> 02:45:13,315 [Thread-12] WARN  processReplacedThreads 
> (ThreadPoolAsynchronousRunner : Task  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@4f8fef92  
> (in deadlocked PoolThread) failed to complete in maximum time  
> 60000ms. Trying interrupt().
> 02:45:13,315 [Thread-12] WARN  processReplacedThreads 
> (ThreadPoolAsynchronousRunner : Task  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@7d792f8e  
> (in deadlocked PoolThread) failed to complete in maximum time  
> 60000ms. Trying interrupt().
> 02:45:13,315 [Thread-12] WARN  processReplacedThreads 
> (ThreadPoolAsynchronousRunner : Task  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@5106f9d  
> (in deadlocked PoolThread) failed to complete in maximum time  
> 60000ms. Trying interrupt().
> 03:08:50,495 [Thread-12] WARN  run(ThreadPoolAsynchronousRunner :  
> com.mchange.v2.async.ThreadPoolAsynchronousRunner 
> $DeadlockDetector@64b2f9b -- APPARENT DEADLOCK!!! Creating  
> emergency threads for unassigned pending tasks!
> 03:08:50,495 [Thread-12] WARN  run(ThreadPoolAsynchronousRunner :  
> com.mchange.v2.async.ThreadPoolAsynchronousRunner 
> $DeadlockDetector@64b2f9b -- APPARENT DEADLOCK!!! Complete Status:  
> [num_managed_threads: 3, num_active: 3; activeTasks:  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@61802f8b  
> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0),  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@40f9afb4  
> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2),  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@23ffaf8d  
> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1);  
> pendingTasks: com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@3ca9ef88]
> 03:09:59,305 [Thread-12] WARN  processReplacedThreads 
> (ThreadPoolAsynchronousRunner : Task  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@61802f8b  
> (in deadlocked PoolThread) failed to complete in maximum time  
> 60000ms. Trying interrupt().
> 03:09:59,305 [Thread-12] WARN  processReplacedThreads 
> (ThreadPoolAsynchronousRunner : Task  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@40f9afb4  
> (in deadlocked PoolThread) failed to complete in maximum time  
> 60000ms. Trying interrupt().
> 03:09:59,305 [Thread-12] WARN  processReplacedThreads 
> (ThreadPoolAsynchronousRunner : Task  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@23ffaf8d  
> (in deadlocked PoolThread) failed to complete in maximum time  
> 60000ms. Trying interrupt().
> 03:50:32,995 [Thread-12] WARN  run(ThreadPoolAsynchronousRunner :  
> com.mchange.v2.async.ThreadPoolAsynchronousRunner 
> $DeadlockDetector@64b2f9b -- APPARENT DEADLOCK!!! Creating  
> emergency threads for unassigned pending tasks!
> 03:50:32,995 [Thread-12] WARN  run(ThreadPoolAsynchronousRunner :  
> com.mchange.v2.async.ThreadPoolAsynchronousRunner 
> $DeadlockDetector@64b2f9b -- APPARENT DEADLOCK!!! Complete Status:  
> [num_managed_threads: 3, num_active: 3; activeTasks:  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@13766fb6  
> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0),  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@3406af92  
> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1),  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@323ef93  
> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2);  
> pendingTasks: com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@4741efb4]
> 03:51:45,975 [Thread-12] WARN  processReplacedThreads 
> (ThreadPoolAsynchronousRunner : Task  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@13766fb6  
> (in deadlocked PoolThread) failed to complete in maximum time  
> 60000ms. Trying interrupt().
> 03:51:45,975 [Thread-12] WARN  processReplacedThreads 
> (ThreadPoolAsynchronousRunner : Task  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@3406af92  
> (in deadlocked PoolThread) failed to complete in maximum time  
> 60000ms. Trying interrupt().
> 03:51:45,975 [Thread-12] WARN  processReplacedThreads 
> (ThreadPoolAsynchronousRunner : Task  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@323ef93  
> (in deadlocked PoolThread) failed to complete in maximum time  
> 60000ms. Trying interrupt().
> 04:32:15,515 [Thread-12] WARN  run(ThreadPoolAsynchronousRunner :  
> com.mchange.v2.async.ThreadPoolAsynchronousRunner 
> $DeadlockDetector@64b2f9b -- APPARENT DEADLOCK!!! Creating  
> emergency threads for unassigned pending tasks!
> 04:32:15,515 [Thread-12] WARN  run(ThreadPoolAsynchronousRunner :  
> com.mchange.v2.async.ThreadPoolAsynchronousRunner 
> $DeadlockDetector@64b2f9b -- APPARENT DEADLOCK!!! Complete Status:  
> [num_managed_threads: 3, num_active: 3; activeTasks:  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@cddefb4  
> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0),  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1f662fb4  
> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1),  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@3e056f94  
> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2);  
> pendingTasks: com.mchange.v2.resourcepool.BasicResourcePool 
> $AcquireTask@4a5e2fb4]
> 04:33:17,635 [Thread-12] WARN  processReplacedThreads 
> (ThreadPoolAsynchronousRunner : Task  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@cddefb4  
> (in deadlocked PoolThread) failed to complete in maximum time  
> 60000ms. Trying interrupt().
> 04:33:17,635 [Thread-12] WARN  processReplacedThreads 
> (ThreadPoolAsynchronousRunner : Task  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1f662fb4  
> (in deadlocked PoolThread) failed to complete in maximum time  
> 60000ms. Trying interrupt().
> 04:33:17,635 [Thread-12] WARN  processReplacedThreads 
> (ThreadPoolAsynchronousRunner : Task  
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@3e056f94  
> (in deadlocked PoolThread) failed to complete in maximum time  
> 60000ms. Trying interrupt().
>
> This is then carried on for the rest of the log, any attempts to  
> contact the system results in a time-out. The system seems to be in  
> a deadlock status, only resolved by rebooting the server.
>
> I will attempt to upgrade to the latest (0.9.1-pre9) version and  
> try to see if the error still occurs, however can you advise of any  
> resolution for this? Could the settings I am using be causing this?
>
> If you require any further detail on this issue, please let me know.
>
> Best regards,
>
> Peter Dolukhanov
> Senior Developer
> Martin Dawes Systems
>
>
> <SystemOut.zip>