[jgroups-users] NAKACK2 xmit-table messages pile-up(memory leak or other issue?)
Brought to you by:
belaban
From: ravi.gawai <rav...@gm...> - 2015-02-21 02:59:03
|
Hi, We have three servers cluster in production with TCP jgroups-replication. Application use ehcache for caching and jgroups for replication. Frameworks versions: ehcache-jgroupsreplication:1.7 *jgroups:3.4.1.Final* Please find tcp-jgroups configuration in file: jgroups_tcp.xml <http://jgroups.1086181.n5.nabble.com/file/n10523/jgroups_tcp.xml> Randomly one of server's NAKACK2 undelivered xmit message counter keep on increasing(xmit_table_undelivered_msgs). This increases heap memory usages and these messages are not released for long time. In 30 mins xmit_table_undelivered_msgs count was reaching like 30k. After an hour these undelivered xmit-table messages pile-up causes memory crash on server. Before server reaching max heap memory we took heap dump and found NAkACK2 taking most of memory. This is happening quit frequently on production from past couple of months. When one server is having issue other two are working fine in terms of replication. I checked tcp dump on problematic server which showed there was communication from-to other servers. This happes randomly to one of server when other two servers works fine. All servers are in same network with same settings. Is there any settings which is not used properly in config which causing this behavior? Is there any setting by which these xmit-messages will get cleared? NakAck2_stats.png <http://jgroups.1086181.n5.nabble.com/file/n10523/NakAck2_stats.png> memoyStats.png <http://jgroups.1086181.n5.nabble.com/file/n10523/memoyStats.png> HeapDump.png <http://jgroups.1086181.n5.nabble.com/file/n10523/HeapDump.png> Thanks, Ravi -- View this message in context: http://jgroups.1086181.n5.nabble.com/NAKACK2-xmit-table-messages-pile-up-memory-leak-or-other-issue-tp10523.html Sent from the JGroups - General mailing list archive at Nabble.com. |