[rvm-research] Possible deadlock in RVMThread code

[Eddie A. <ea...@cs...>]

Hi all,

I have been trying to implement a concurrent garbage collector in Jikes 
RVM, so I have been looking closely at the threading code.  I think I've 
run into a deadlock while testing my collector.

In RVMThread.java, there is a static Monitor handshakeLock that prevents 
multiple handshakes from occurring at the same time.  This lock also 
protects the communicationLockBySlot array.

In RVMThread.assignThreadSlot(), we have this code:

handshakeLock.lockWithHandshake();
if (communicationLockBySlot[threadSlot] == null) {
   communicationLockBySlot[threadSlot] = new Monitor();
}
handshakeLock.unlock();

"new Monitor()" allocates, which can cause a GC.  If it does cause a GC, 
it does not release the handshakeLock, so the GC thread cannot acquire 
the handshakeLock in hardHandshakeSuspend() and the GC cannot proceed. 
I believe this is what is happening in my situation.

Does my reasoning look correct?  Any suggestions for how to fix this?

Thanks,
Eddie