Re: [jgroups-users] NAKACK2 xmit-table messages pile-up(memory leak or other issue?)

[Bela B. <be...@ya...>]

You said this system is running in production: have these errors 
occurred before, or did they start a couple of months ago ? If the 
latter, what did you change ?

Is it the same physical box that's failing, or different boxes ?

I can see *a lot* of retransmission (xmit_reqs_received/sent) going on; 
this points to a flaky network.

Why do you use DAISYCHAIN ? This is experimental at best; I suggest test 
without it.

Can you try 3.6.2 instead of 3.4.1 to see if the issue still occurs ?

On 21/02/15 03:58, ravi.gawai wrote:
> Hi,
>      We have three servers cluster in production with TCP
> jgroups-replication. Application use ehcache  for caching and jgroups for
> replication.
>
> Frameworks versions: ehcache-jgroupsreplication:1.7
> *jgroups:3.4.1.Final*
> Please find tcp-jgroups configuration in file:  jgroups_tcp.xml
> <http://jgroups.1086181.n5.nabble.com/file/n10523/jgroups_tcp.xml>
>
> Randomly one of server's NAKACK2 undelivered xmit message counter keep on
> increasing(xmit_table_undelivered_msgs). This increases heap memory usages
> and these messages are not released for long time. In 30 mins
> xmit_table_undelivered_msgs count was reaching like 30k.
>
> After an hour these undelivered xmit-table messages pile-up causes memory
> crash on server. Before server reaching max heap memory we took heap dump
> and found NAkACK2 taking most of memory.
>
> This is happening quit frequently on production from past couple of months.
> When one server is having issue other two are working fine in terms of
> replication. I checked tcp dump on problematic server which showed there was
> communication from-to other servers.
>
> This happes randomly to one of server when other two servers works fine. All
> servers are in same network with same settings.
>
> Is there any settings which is not used properly in config which causing
> this behavior? Is there any setting by which these xmit-messages will get
> cleared?
>
>    NakAck2_stats.png
> <http://jgroups.1086181.n5.nabble.com/file/n10523/NakAck2_stats.png>
> memoyStats.png
> <http://jgroups.1086181.n5.nabble.com/file/n10523/memoyStats.png>
> HeapDump.png <http://jgroups.1086181.n5.nabble.com/file/n10523/HeapDump.png>
>
> Thanks,
> Ravi
>
>
>
> --
> View this message in context: http://jgroups.1086181.n5.nabble.com/NAKACK2-xmit-table-messages-pile-up-memory-leak-or-other-issue-tp10523.html
> Sent from the JGroups - General mailing list archive at Nabble.com.


-- 
Bela Ban, JGroups lead (http://www.jgroups.org)