[Bigdata-developers] Design patterns for shutdown of embedded services and in particular for the Journal.

[Bryan T. <br...@sy...>]

We currently have a deadlock which occasionally arises in Journal#shutdownNow().  That method closes down various services used by the journal (an executor service, the concurrency manager (which also shuts down the write service), and the local transaction manager).  If there is a concurrent commit running, then the code will occasionally deadlock as illustrated below.  

I'd like to find someone with experience in shutdown patterns for embedded services who could help out on this shutdown pattern.  Any volunteers?

Bryan

"pool-1-thread-560" daemon prio=10 tid=0x52e9ac00 nid=0x7ae6 waiting on condition [0x41869000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x88a1fe40> (a java.util.concurrent.FutureTask$Sync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:905)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1217)
	at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218)
	at java.util.concurrent.FutureTask.get(FutureTask.java:83)
	at com.bigdata.journal.JournalTransactionService.commitImpl(JournalTransactionService.java:300)
	at com.bigdata.service.AbstractTransactionService.commit(AbstractTransactionService.java:1626)
	at com.bigdata.journal.Journal.commit(Journal.java:726)

"main" prio=10 tid=0x08f62c00 nid=0x6c29 waiting on condition [0xb7395000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x88a241b8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
	at com.bigdata.service.AbstractTransactionService.shutdownNow(AbstractTransactionService.java:488)
	at com.bigdata.journal.Journal$1.shutdownNow(Journal.java:201)
	at com.bigdata.journal.Journal.shutdownNow(Journal.java:852)
	- locked <0x86a5dc40> (a com.bigdata.journal.Journal)