[javagroups-users] threadlock problem

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

Hi, I'm using JGroups 2.2.7, with the default-minimalthreads.xml 
configuration (converted to a property string).  I've got a 
DistributedHashTable running on top of it, and it randomly deadlocks 
during a put() operation.  The pertinant part of the stack trace 
follows... any help would be appreciated.

Thanks,
Gary Herndon

--- begin config ---
UDP(mcast_addr=228.8.8.8;mcast_port=45567;mcast_recv_buf_size=64000;\
mcast_send_buf_size=32000;down_thread=false;ip_mcast=true;\
use_incoming_packet_handler=true;use_outgoing_packet_handler=true;\
ucast_recv_buf_size=64000;ucast_send_buf_size=32000;ip_ttl=32;\
loopback=true):\
AUTOCONF(down_thread=false;up_thread=false):\
PING(timeout=2000;num_initial_members=3;down_thread=false;up_thread=false):\
MERGE2(min_interval=5000;max_interval=10000;down_thread=false;up_thread=false):\
FD(timeout=2000;max_tries=3;down_thread=false;up_thread=false;shun=true):\
VERIFY_SUSPECT(timeout=1500;down_thread=false;up_thread=false):\
pbcast.NAKACK(max_xmit_size=8192;use_mcast_xmit=false;gc_lag=50;\
retransmit_timeout=600,1200,2400,4800;down_thread=false;up_thread=false):\
UNICAST(timeout=1200,2400,3600;down_thread=false;up_thread=false):\
pbcast.STABLE(stability_delay=1000;desired_avg_gossip=20000;\
max_bytes=0;down_thread=false;up_thread=false):\
FRAG(frag_size=8192;down_thread=false;up_thread=false):\
pbcast.GMS(join_timeout=3000;join_retry_timeout=2000;shun=true;\
print_local_addr=false;down_thread=false;up_thread=false):\
pbcast.STATE_TRANSFER(down_thread=false;up_thread=false)
--- end config ---

--- begin stack trace snippet ---

Found one Java-level deadlock:
=============================
"TimeScheduler.Thread":
   waiting to lock monitor 0x000e02f8 (object 0xcd44c490, a 
java.util.HashMap),
   which is held by "MessageDispatcher up processing thread"
"MessageDispatcher up processing thread":
   waiting to lock monitor 0x001ac1c8 (object 0xcd6b2d58, a 
org.jgroups.stack.Retransmitter$Entry),
   which is held by "TimeScheduler.Thread"

Java stack information for the threads listed above:
===================================================
"TimeScheduler.Thread":
         at org.jgroups.stack.AckSenderWindow.ack(AckSenderWindow.java:165)
         - waiting to lock <0xcd44c490> (a java.util.HashMap)
         at 
org.jgroups.protocols.UNICAST.handleAckReceived(UNICAST.java:391)
         at org.jgroups.protocols.UNICAST.up(UNICAST.java:171)
         at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:437)
         at org.jgroups.stack.Protocol.passUp(Protocol.java:491)
         at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:389)
         at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:437)
         at org.jgroups.stack.Protocol.passUp(Protocol.java:491)
         at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:120)
         at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:437)
         at org.jgroups.stack.Protocol.passUp(Protocol.java:491)
         at org.jgroups.protocols.FD.up(FD.java:242)
         at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:437)
         at org.jgroups.stack.Protocol.passUp(Protocol.java:491)
         at org.jgroups.protocols.MERGE2.up(MERGE2.java:135)
         at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:437)
         at org.jgroups.stack.Protocol.passUp(Protocol.java:491)
         at org.jgroups.protocols.PING.up(PING.java:190)
         at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:437)
         at org.jgroups.stack.Protocol.passUp(Protocol.java:491)
         at org.jgroups.stack.Protocol.up(Protocol.java:528)
         at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:437)
         at org.jgroups.stack.Protocol.passUp(Protocol.java:491)
         at org.jgroups.protocols.UDP.sendUdpMessage(UDP.java:774)
         at org.jgroups.protocols.UDP.down(UDP.java:624)
         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:467)
         at org.jgroups.stack.Protocol.passDown(Protocol.java:510)
         at org.jgroups.stack.Protocol.down(Protocol.java:540)
         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:467)
         at org.jgroups.stack.Protocol.passDown(Protocol.java:510)
         at org.jgroups.protocols.PING.down(PING.java:423)
         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:467)
         at org.jgroups.stack.Protocol.passDown(Protocol.java:510)
         at org.jgroups.protocols.MERGE2.down(MERGE2.java:170)
         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:467)
         at org.jgroups.stack.Protocol.passDown(Protocol.java:510)
         at org.jgroups.protocols.FD.down(FD.java:278)
         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:467)
         at org.jgroups.stack.Protocol.passDown(Protocol.java:510)
         at org.jgroups.stack.Protocol.down(Protocol.java:540)
         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:467)
         at org.jgroups.stack.Protocol.passDown(Protocol.java:510)
         at org.jgroups.protocols.pbcast.NAKACK.down(NAKACK.java:297)
         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:467)
         at org.jgroups.stack.Protocol.passDown(Protocol.java:510)
         at org.jgroups.protocols.UNICAST.sendAck(UNICAST.java:400)
         at org.jgroups.protocols.UNICAST.up(UNICAST.java:167)
         at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:437)
         at org.jgroups.stack.Protocol.passUp(Protocol.java:491)
         at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:389)
         at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:437)
         at org.jgroups.stack.Protocol.passUp(Protocol.java:491)
         at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:120)
         at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:437)
         at org.jgroups.stack.Protocol.passUp(Protocol.java:491)
         at org.jgroups.protocols.FD.up(FD.java:242)
         at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:437)
         at org.jgroups.stack.Protocol.passUp(Protocol.java:491)
         at org.jgroups.protocols.MERGE2.up(MERGE2.java:135)
         at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:437)
         at org.jgroups.stack.Protocol.passUp(Protocol.java:491)
         at org.jgroups.protocols.PING.up(PING.java:190)
         at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:437)
         at org.jgroups.stack.Protocol.passUp(Protocol.java:491)
         at org.jgroups.stack.Protocol.up(Protocol.java:528)
         at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:437)
         at org.jgroups.stack.Protocol.passUp(Protocol.java:491)
         at org.jgroups.protocols.UDP.sendUdpMessage(UDP.java:774)
         at org.jgroups.protocols.UDP.down(UDP.java:624)
         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:467)
         at org.jgroups.stack.Protocol.passDown(Protocol.java:510)
         at org.jgroups.stack.Protocol.down(Protocol.java:540)
         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:467)
         at org.jgroups.stack.Protocol.passDown(Protocol.java:510)
         at org.jgroups.protocols.PING.down(PING.java:423)
         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:467)
         at org.jgroups.stack.Protocol.passDown(Protocol.java:510)
         at org.jgroups.protocols.MERGE2.down(MERGE2.java:170)
         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:467)
         at org.jgroups.stack.Protocol.passDown(Protocol.java:510)
         at org.jgroups.protocols.FD.down(FD.java:278)
         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:467)
         at org.jgroups.stack.Protocol.passDown(Protocol.java:510)
         at org.jgroups.stack.Protocol.down(Protocol.java:540)
         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:467)
         at org.jgroups.stack.Protocol.passDown(Protocol.java:510)
         at org.jgroups.protocols.pbcast.NAKACK.down(NAKACK.java:297)
         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:467)
         at org.jgroups.stack.Protocol.passDown(Protocol.java:510)
         at org.jgroups.protocols.UNICAST.retransmit(UNICAST.java:325)
         at 
org.jgroups.stack.AckSenderWindow.retransmit(AckSenderWindow.java:208)
         at 
org.jgroups.stack.Retransmitter$Entry.run(Retransmitter.java:333)
         - locked <0xcd6b2d58> (a org.jgroups.stack.Retransmitter$Entry)
         at org.jgroups.util.TimeScheduler._run(TimeScheduler.java:401)
         at org.jgroups.util.TimeScheduler.access$000(TimeScheduler.java:46)
         at org.jgroups.util.TimeScheduler$Loop.run(TimeScheduler.java:135)
         at java.lang.Thread.run(Unknown Source)
"MessageDispatcher up processing thread":
         at org.jgroups.stack.Retransmitter.remove(Retransmitter.java:126)
         - waiting to lock <0xcd6b2d58> (a 
org.jgroups.stack.Retransmitter$Entry)
         - locked <0xcd44bc90> (a java.util.LinkedList)
         at org.jgroups.stack.AckSenderWindow.ack(AckSenderWindow.java:167)
         - locked <0xcd44c490> (a java.util.HashMap)
         at 
org.jgroups.protocols.UNICAST.handleAckReceived(UNICAST.java:391)
         at org.jgroups.protocols.UNICAST.up(UNICAST.java:171)
         at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:437)
         at org.jgroups.stack.Protocol.passUp(Protocol.java:491)
         at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:389)
         at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:437)
         at org.jgroups.stack.Protocol.passUp(Protocol.java:491)
         at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:120)
         at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:437)
         at org.jgroups.stack.Protocol.passUp(Protocol.java:491)
         at org.jgroups.protocols.FD.up(FD.java:242)
         at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:437)
         at org.jgroups.stack.Protocol.passUp(Protocol.java:491)
         at org.jgroups.protocols.MERGE2.up(MERGE2.java:135)
         at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:437)
         at org.jgroups.stack.Protocol.passUp(Protocol.java:491)
         at org.jgroups.protocols.PING.up(PING.java:190)
         at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:437)
         at org.jgroups.stack.Protocol.passUp(Protocol.java:491)
         at org.jgroups.stack.Protocol.up(Protocol.java:528)
         at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:437)
         at org.jgroups.stack.Protocol.passUp(Protocol.java:491)
         at org.jgroups.protocols.UDP.sendUdpMessage(UDP.java:774)
         at org.jgroups.protocols.UDP.down(UDP.java:624)
         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:467)
         at org.jgroups.stack.Protocol.passDown(Protocol.java:510)
         at org.jgroups.stack.Protocol.down(Protocol.java:540)
         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:467)
         at org.jgroups.stack.Protocol.passDown(Protocol.java:510)
         at org.jgroups.protocols.PING.down(PING.java:423)
         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:467)
         at org.jgroups.stack.Protocol.passDown(Protocol.java:510)
         at org.jgroups.protocols.MERGE2.down(MERGE2.java:170)
         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:467)
         at org.jgroups.stack.Protocol.passDown(Protocol.java:510)
         at org.jgroups.protocols.FD.down(FD.java:278)
         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:467)
         at org.jgroups.stack.Protocol.passDown(Protocol.java:510)
         at org.jgroups.stack.Protocol.down(Protocol.java:540)
         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:467)
         at org.jgroups.stack.Protocol.passDown(Protocol.java:510)
         at org.jgroups.protocols.pbcast.NAKACK.down(NAKACK.java:297)
         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:467)
         at org.jgroups.stack.Protocol.passDown(Protocol.java:510)
         at org.jgroups.protocols.UNICAST.sendAck(UNICAST.java:400)
         at org.jgroups.protocols.UNICAST.up(UNICAST.java:167)
         at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:437)
         at org.jgroups.stack.Protocol.passUp(Protocol.java:491)
         at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:389)
         at org.jgroups.stack.Protocol.passUp(Protocol.java:491)
         at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:120)
         at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:437)
         at org.jgroups.stack.Protocol.passUp(Protocol.java:491)
         at org.jgroups.protocols.FD.up(FD.java:242)
         at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:437)
         at org.jgroups.stack.Protocol.passUp(Protocol.java:491)
         at org.jgroups.protocols.MERGE2.up(MERGE2.java:135)
         at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:437)
         at org.jgroups.stack.Protocol.passUp(Protocol.java:491)
         at org.jgroups.protocols.PING.up(PING.java:190)
         at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:437)
         at org.jgroups.stack.Protocol.passUp(Protocol.java:491)
         at org.jgroups.stack.Protocol.up(Protocol.java:528)
         at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:437)
         at org.jgroups.stack.Protocol.passUp(Protocol.java:491)
         at org.jgroups.protocols.UDP.sendUdpMessage(UDP.java:774)
         at org.jgroups.protocols.UDP.down(UDP.java:624)
         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:467)
         at org.jgroups.stack.Protocol.passDown(Protocol.java:510)
         at org.jgroups.stack.Protocol.down(Protocol.java:540)
         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:467)
         at org.jgroups.stack.Protocol.passDown(Protocol.java:510)
         at org.jgroups.protocols.PING.down(PING.java:423)
         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:467)
         at org.jgroups.stack.Protocol.passDown(Protocol.java:510)
         at org.jgroups.protocols.MERGE2.down(MERGE2.java:170)
         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:467)
         at org.jgroups.stack.Protocol.passDown(Protocol.java:510)
         at org.jgroups.protocols.FD.down(FD.java:278)
         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:467)
         at org.jgroups.stack.Protocol.passDown(Protocol.java:510)
         at org.jgroups.stack.Protocol.down(Protocol.java:540)
         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:467)
         at org.jgroups.stack.Protocol.passDown(Protocol.java:510)
         at org.jgroups.protocols.pbcast.NAKACK.down(NAKACK.java:297)
         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:467)
         at org.jgroups.stack.Protocol.passDown(Protocol.java:510)
         at 
org.jgroups.stack.AckSenderWindow.addMessage(AckSenderWindow.java:223)
         at org.jgroups.stack.AckSenderWindow.add(AckSenderWindow.java:133)
         - locked <0xcd44c490> (a java.util.HashMap)
         at org.jgroups.protocols.UNICAST.down(UNICAST.java:229)
         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:467)
         at org.jgroups.stack.Protocol.passDown(Protocol.java:510)
         at org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:292)
         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:467)
         at org.jgroups.stack.Protocol.passDown(Protocol.java:510)
         at org.jgroups.protocols.FRAG.down(FRAG.java:145)
         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:467)
         at org.jgroups.stack.Protocol.passDown(Protocol.java:510)
         at org.jgroups.protocols.pbcast.GMS.down(GMS.java:626)
         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:467)
         at org.jgroups.stack.Protocol.passDown(Protocol.java:510)
         at 
org.jgroups.protocols.pbcast.STATE_TRANSFER.down(STATE_TRANSFER.java:217)
         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:467)
         at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:331)
         at org.jgroups.JChannel.down(JChannel.java:1035)
         at 
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:733)
         at 
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.passDown(MessageDispatcher.java:660)
         at 
org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:655)
         at 
org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:512)
         at 
org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:326)
         at 
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUp(MessageDispatcher.java:722)
         at 
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.access$300(MessageDispatcher.java:554)
         at 
org.jgroups.blocks.MessageDispatcher$1.run(MessageDispatcher.java:691)
         at java.lang.Thread.run(Unknown Source)

Found 1 deadlock.
--- end snippet ---