Re: [Wrapper-user] JVM appears hung: Timed out waiting for signalfrom JVM

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

A high CPU usage of a Java application is most of the time caused by the Garbage Collector. Of course, only if it is not a calculation intensive program. The GC is the reason we see ping timeouts on our Java based server. Leif, is there any possibility to set the ping timeout values to a reasonable value so the Java VM can survive a 1min long Full GC pause occuring about once a day? I don't want to set the ping timeout that high, as then during normal usage the error detection via ping does not work anymore.

> -----Original Message-----
> From: wra...@li... 
> [mailto:wra...@li...] On Behalf 
> Of Leif Mortenson
> Sent: Wednesday, July 23, 2008 9:46 AM
> To: wra...@li...
> Subject: Re: [Wrapper-user] JVM appears hung: Timed out 
> waiting for signalfrom JVM
> 
> Patrick,
> I agree that it does not appear to be doing you any good.   I am
> wondering why you are needing to play with it it all thought.
> 
> What is the memory usage of your application vs the physical memory of
> the machine?  Is this being caused by swapping?  Does your application
> modify the thread priority of any of its threads?   Are other
> applications consuming lots of CPU?
> 
> It may be possible to improve the way the Wrapper operates under these
> conditions if I can learn exactly what is happening.
> 
> Thanks,
> Leif
> 
> On Wed, Jul 23, 2008 at 3:52 AM,  <PFa...@qu...> wrote:
> > Leif,
> >
> > Thank you for your reply.  We will schedule the 
> configuration changes as
> > part of our next release.
> >
> > Assuming that we continue to experience the same problem in 
> the future, is
> > there any reason why we would want to continue messing 
> around with the value
> > of the property (wrapper.ping.timeout) instead of turning off the
> > functionality altogether by setting it to zero?  I guess I 
> don't see what
> > benefit it is providing when we are not using the restart 
> functionality.
> >
> > Patrick M. Farrell, PMP, CSM
> > Consultant Mentor
> > Quick Solutions, Inc.
> > A four time INC 500 winner!
> > pfa...@qu...
> >
> > -----wra...@li... wrote: -----
> >
> > To: wra...@li...
> > From: "Leif Mortenson" <le...@ta...>
> > Sent by: wra...@li...
> > Date: 07/22/2008 01:27PM
> > Subject: Re: [Wrapper-user] JVM appears hung: Timed out 
> waiting for signal
> > from JVM
> >
> > Patrick,
> > If you are having ping timeout problems, it is usually caused by the
> > application using more memory than the system has available in
> > Physical Memory.  Java can get 100-1000x slower when the OS 
> does even
> > a little bit of swapping of its memory space.  Unless you 
> are changing
> > thread priorities in your application, the Wrapper will 
> generally not
> > lose the ability to respond to pings even when the CPU 
> usage is fairly
> > high.
> >
> > Regardless, the property needed to extend the amount of time the
> > Wrapper will wait for the JVM is as you mentioned:
> > wrapper.ping.timeout
> >
> > You might want to try setting wrapper.debug=true.  This will let you
> > see exactly when pings are send and responded to.
> >
> > If you are still getting the timeouts with a ping timeout 
> of 900 then
> > the JVM must be eating too much serial for more than 15minutes.
> >
> > As a test, you might want to try setting the ping timeout of 1800.
> >
> > Cheers,
> > Leif
> >
> > On Wed, Jul 23, 2008 at 2:18 AM,  
> <PFa...@qu...> wrote:
> >> Hello,
> >>
> >> The purpose of this post is to determine the most 
> appropriate course of
> >> action given our current situation.
> >>
> >> Background:
> >> The application is running the 3.3.0 x86 release of the 
> Java Service
> >> Wrapper
> >> and using the WrapperSimpleApp main class at startup.  The 
> supporting
> >> hardware has 8 CPUs running 4 instances of the application.  The
> >> processing
> >> for this service is quite CPU and memory intensive.  It is 
> not uncommon
> >> for
> >> the CPU utilization on the box to reach 100% for 3-5 
> minutes at a time,
> >> and
> >> if four large files arrived in quick succession, the CPU 
> utilization could
> >> potentially be pegged for as long as 15 minutes.
> >>
> >> Symptom:
> >> The Java Service Wrapper is reporting that the JVM appears 
> to be hung and
> >> that the timeout expired while waiting on a signal from 
> the JVM.  The Java
> >> Service Wrapper then shuts down the JVM even though we believe the
> >> application is processing normally.  Note:  We have 
> explicitly disabled
> >> the
> >> automatic restart functionality as we would like our 
> existing monitoring
> >> software to detect and report the problem with the instance of the
> >> application.  The following snippet is from the wrapper.log file:
> >>
> >> ERROR  | wrapper  | 2008/07/21 16:15:53 | JVM appears 
> hung: Timed out
> >> waiting for signal from JVM.
> >> STATUS | wrapper  | 2008/07/21 16:15:54 | JVM Restarts 
> disabled.  Shutting
> >> down.
> >> STATUS | wrapper  | 2008/07/21 16:15:54 | <-- Wrapper Stopped
> >>
> >> The wrapper configuration is as follows:
> >>
> >> # -----------------------------------------------------------------
> >> # This file contains the configuration for the Java Service Wrapper
> >> # -----------------------------------------------------------------
> >>
> >> # Set the name of the service.
> >> wrapper.name=VPC Gen2 Capture Service 1
> >>
> >> # Set the location of the wrapper log
> >> wrapper.logfile=../logs/wrapper.1.log
> >>
> >> # Set the java command line options.
> >> wrapper.java.additional.1=-DFILE_PREFIX=//localhost/data
> >> wrapper.java.additional.2=-Dlog4j.configuration=log4j.1.xml
> >>
> >> 
> wrapper.java.additional.3=-DCAPTURE_SERVICES_LOG_DIR=D:\vpc\ca
> pture-services\
> >>
> >> 
> wrapper.java.additional.4=-Dcom.sun.management.jmxremote.authe
> nticate=false
> >> wrapper.java.additional.5=-Dcom.sun.management.jmxremote.port=1151
> >> wrapper.java.additional.6=-Dcom.sun.management.jmxremote.ssl=false
> >> wrapper.java.additional.7=-Xmx1150m
> >> wrapper.java.additional.8=-Xms512m
> >> wrapper.java.additional.9=-XX:MaxPermSize=192m
> >> wrapper.java.additional.10=-server
> >>
> >> # Set the application classpath.
> >> # The application classpath must contain wrapper.jar!
> >> wrapper.java.classpath.1=../lib/wrapper.jar
> >> wrapper.java.classpath.2=../../lib/vpc-framework-1.1.3.jar
> >> wrapper.java.classpath.3=../../lib/capture-services-1.1.3.jar
> >>
> >> # -----------------------------------------------------------------
> >> # The following properties should not need to be modified.
> >> # -----------------------------------------------------------------
> >>
> >> # Set the application classname and optionally any
> >> # command-line parameters that must be passed.
> >> wrapper.app.parameter.1=com.jpmc.vpc.core.StartApplication
> >>
> >> # Set the Java Library Path.
> >> wrapper.java.library.path.1=../lib
> >>
> >> # Set the Java command.
> >> wrapper.java.command=%JAVA_HOME%/bin/java
> >>
> >> # Set the Java Service Wrapper main class.
> >> wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperSimpleApp
> >>
> >> # The following property instructs the Java Service Wrapper
> >> # to append the system PATH to the java.library.path at
> >> # runtime.  This is used in order to load secondary DLLs.
> >> wrapper.java.library.path.append_system_path=true
> >>
> >> # Set the debug flag.
> >> wrapper.debug=false
> >>
> >> # Allow 5 minutes between pings
> >> wrapper.ping.interval=300
> >>
> >> # Do NOT try to restart the JVM
> >> wrapper.disable_restarts=TRUE
> >>
> >> # Number of seconds to wait for the ping to timeout
> >> wrapper.ping.timeout=900
> >>
> >> Our current thought is that we could set the value of the
> >> wrapper.ping.timeout property to be zero.  This should 
> prevent the Java
> >> Service Wrapper from attempting to ping the WrapperSimpleApp and
> >> ultimately
> >> prevent the false detection of a hung JVM which was 
> causing the JVM to be
> >> shut down in the middle of normal processing.  Does that 
> sound about
> >> accurate?
> >>
> >> Has anyone else out there experienced this or a similar 
> problem?  If so,
> >> what was your solution?  Is there an alternate property 
> that I should be
> >> focused on instead of the wrapper.ping.timeout property?  
> I am aware of
> >> the
> >> wrapper.cpu.timeout property, but I have not seen any 
> messages in the log
> >> to
> >> indicate that the wrapper has not received any CPU time 
> for N seconds.
> >>
> >> I would appreciate any input that you may have to offer.
> >>
> >> Patrick M. Farrell, PMP, CSM
> >> Consultant Mentor
> >> Quick Solutions, Inc.
> >> A four time INC 500 winner!
> >> pfa...@qu...
> >>
> >> 
> --------------------------------------------------------------
> -----------
> >> This SF.Net email is sponsored by the Moblin Your Move Developer's
> >> challenge
> >> Build the coolest Linux based applications with Moblin SDK 
> & win great
> >> prizes
> >> Grand prize is a trip for two to an Open Source event 
> anywhere in the
> >> world
> >> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> >> _______________________________________________
> >> Wrapper-user mailing list
> >> Wra...@li...
> >> https://lists.sourceforge.net/lists/listinfo/wrapper-user
> >>
> >>
> >
> > 
> --------------------------------------------------------------
> -----------
> > This SF.Net email is sponsored by the Moblin Your Move 
> Developer's challenge
> > Build the coolest Linux based applications with Moblin SDK 
> & win great
> > prizes
> > Grand prize is a trip for two to an Open Source event 
> anywhere in the world
> > http://moblin-contest.org/redirect.php?banner_id=100&url=/
> > _______________________________________________
> > Wrapper-user mailing list
> > Wra...@li...
> > https://lists.sourceforge.net/lists/listinfo/wrapper-user
> >
> >
> > 
> --------------------------------------------------------------
> -----------
> > This SF.Net email is sponsored by the Moblin Your Move 
> Developer's challenge
> > Build the coolest Linux based applications with Moblin SDK 
> & win great
> > prizes
> > Grand prize is a trip for two to an Open Source event 
> anywhere in the world
> > http://moblin-contest.org/redirect.php?banner_id=100&url=/
> > _______________________________________________
> > Wrapper-user mailing list
> > Wra...@li...
> > https://lists.sourceforge.net/lists/listinfo/wrapper-user
> >
> >
> 
> --------------------------------------------------------------
> -----------
> This SF.Net email is sponsored by the Moblin Your Move 
> Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & 
> win great prizes
> Grand prize is a trip for two to an Open Source event 
> anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Wrapper-user mailing list
> Wra...@li...
> https://lists.sourceforge.net/lists/listinfo/wrapper-user
> 
> 