Re: [javagroups-users] jgroups blocks after outofmemory

[Bela B. <be...@ya...>]

You should really use 2.12.1 as 2.8 is unsupported [1].

I've improved handling of temporary OOM situations in more recent 
releases, but running a JGroups instance out of memory and then 
expecting it to continue functionaing properly isn't something I've 
tested extensively.

Needless to say that this is probably the case for other systems as well.

[1] http://community.jboss.org/wiki/Support



On 09/21/2011 04:43 PM, Dejan Pecar wrote:
> hi,
>
> we're using jgroups 2.8.0 with the attached config.
>
> scenario is the following:
>
> vm1 and vm2 attached to the same channel.
>
> vm1 gets a job an goes to the limit of its available memory only about
> 20kb left of 256mb but still ok.
>
> now vm3 gets attached to the channel. this causes now that the vm1
> (which is running on limit) gets
> an OutOfMemoryError in TP.java but the vm is still up and running
> because jgroups catches Throwable (which is a very bad idea).
>
> (in the logfile this looks like this then:
>
> 21.09.2011 10:15:36 org.jgroups.logging.JDKLogImpl error
> SCHWERWIEGEND: failed handling incoming message
> java.lang.OutOfMemoryError: Java heap space
> 21.09.2011 10:15:36 org.jgroups.logging.JDKLogImpl error
> SCHWERWIEGEND: failed handling incoming message
> java.lang.OutOfMemoryError: Java heap space
> 21.09.2011 10:15:40 org.jgroups.logging.JDKLogImpl error
> SCHWERWIEGEND: failed retransmission task
> java.lang.OutOfMemoryError: Java heap space
> 21.09.2011 10:15:39 org.jgroups.logging.JDKLogImpl error
> SCHWERWIEGEND: failed handling incoming message
> java.lang.OutOfMemoryError: Java heap space
> 21.09.2011 10:15:39 org.jgroups.logging.JDKLogImpl error
> SCHWERWIEGEND: failed sending discovery request
> java.lang.OutOfMemoryError: Java heap space
> 21.09.2011 10:15:39 org.jgroups.logging.JDKLogImpl error
> SCHWERWIEGEND: failed handling incoming message
> java.lang.OutOfMemoryError: Java heap space
> 21.09.2011 10:15:38 org.jgroups.logging.JDKLogImpl error
> SCHWERWIEGEND: failed handling incoming message
> java.lang.OutOfMemoryError: Java heap space
> 21.09.2011 10:15:40 org.jgroups.logging.JDKLogImpl error
> SCHWERWIEGEND: uncaught exception in Thread[multicast
> receiver,AbacusUserLoginGr
> oupCom_pecar-z600I1YUDUI2WMAA4T4EAIJGLU21IC,pecar-z600-26985,5,JGroups]
> (thread
> group=org.jgroups.util.Util$1[name=JGroups,maxpri=10] )
> java.lang.OutOfMemoryError: Java heap space
> 21.09.2011 10:15:40 org.jgroups.logging.JDKLogImpl error
> SCHWERWIEGEND: failed retransmission task
> java.lang.OutOfMemoryError: Java heap space
> 21.09.2011 10:15:41 org.jgroups.logging.JDKLogImpl error
> SCHWERWIEGEND: failed handling incoming message
> java.lang.OutOfMemoryError: Java heap space)
>
> so this vm1 has no chance to detect that something is wrong. the job
> finishes successfully because the
> memory for the job was enough. the job is now finished there is plenty
> of memory again but jgoups is
> still out of order and all vms attached to this channel are also blocked
> until i kill all vms attached to the channel manually.
> this is the worst case sometimes its enough to only restart the vm which
> caused the OutOfMemoryError and the other channel members
> can regenerate.
>
> any solution for this problem anyone ?
>
> regards
> dejan


-- 
Bela Ban
Lead JGroups / Clustering Team
JBoss