Re: [Wrapper-user] JVM appears hung: Timed out waiting for signal from JVM

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Patrick,
If you are having ping timeout problems, it is usually caused by the
application using more memory than the system has available in
Physical Memory.  Java can get 100-1000x slower when the OS does even
a little bit of swapping of its memory space.  Unless you are changing
thread priorities in your application, the Wrapper will generally not
lose the ability to respond to pings even when the CPU usage is fairly
high.

Regardless, the property needed to extend the amount of time the
Wrapper will wait for the JVM is as you mentioned:
wrapper.ping.timeout

You might want to try setting wrapper.debug=true.  This will let you
see exactly when pings are send and responded to.

If you are still getting the timeouts with a ping timeout of 900 then
the JVM must be eating too much serial for more than 15minutes.

As a test, you might want to try setting the ping timeout of 1800.

Cheers,
Leif

On Wed, Jul 23, 2008 at 2:18 AM,  <PFa...@qu...> wrote:
> Hello,
>
> The purpose of this post is to determine the most appropriate course of
> action given our current situation.
>
> Background:
> The application is running the 3.3.0 x86 release of the Java Service Wrapper
> and using the WrapperSimpleApp main class at startup.  The supporting
> hardware has 8 CPUs running 4 instances of the application.  The processing
> for this service is quite CPU and memory intensive.  It is not uncommon for
> the CPU utilization on the box to reach 100% for 3-5 minutes at a time, and
> if four large files arrived in quick succession, the CPU utilization could
> potentially be pegged for as long as 15 minutes.
>
> Symptom:
> The Java Service Wrapper is reporting that the JVM appears to be hung and
> that the timeout expired while waiting on a signal from the JVM.  The Java
> Service Wrapper then shuts down the JVM even though we believe the
> application is processing normally.  Note:  We have explicitly disabled the
> automatic restart functionality as we would like our existing monitoring
> software to detect and report the problem with the instance of the
> application.  The following snippet is from the wrapper.log file:
>
> ERROR  | wrapper  | 2008/07/21 16:15:53 | JVM appears hung: Timed out
> waiting for signal from JVM.
> STATUS | wrapper  | 2008/07/21 16:15:54 | JVM Restarts disabled.  Shutting
> down.
> STATUS | wrapper  | 2008/07/21 16:15:54 | <-- Wrapper Stopped
>
> The wrapper configuration is as follows:
>
> # -----------------------------------------------------------------
> # This file contains the configuration for the Java Service Wrapper
> # -----------------------------------------------------------------
>
> # Set the name of the service.
> wrapper.name=VPC Gen2 Capture Service 1
>
> # Set the location of the wrapper log
> wrapper.logfile=../logs/wrapper.1.log
>
> # Set the java command line options.
> wrapper.java.additional.1=-DFILE_PREFIX=//localhost/data
> wrapper.java.additional.2=-Dlog4j.configuration=log4j.1.xml
> wrapper.java.additional.3=-DCAPTURE_SERVICES_LOG_DIR=D:\vpc\capture-services\
> wrapper.java.additional.4=-Dcom.sun.management.jmxremote.authenticate=false
> wrapper.java.additional.5=-Dcom.sun.management.jmxremote.port=1151
> wrapper.java.additional.6=-Dcom.sun.management.jmxremote.ssl=false
> wrapper.java.additional.7=-Xmx1150m
> wrapper.java.additional.8=-Xms512m
> wrapper.java.additional.9=-XX:MaxPermSize=192m
> wrapper.java.additional.10=-server
>
> # Set the application classpath.
> # The application classpath must contain wrapper.jar!
> wrapper.java.classpath.1=../lib/wrapper.jar
> wrapper.java.classpath.2=../../lib/vpc-framework-1.1.3.jar
> wrapper.java.classpath.3=../../lib/capture-services-1.1.3.jar
>
> # -----------------------------------------------------------------
> # The following properties should not need to be modified.
> # -----------------------------------------------------------------
>
> # Set the application classname and optionally any
> # command-line parameters that must be passed.
> wrapper.app.parameter.1=com.jpmc.vpc.core.StartApplication
>
> # Set the Java Library Path.
> wrapper.java.library.path.1=../lib
>
> # Set the Java command.
> wrapper.java.command=%JAVA_HOME%/bin/java
>
> # Set the Java Service Wrapper main class.
> wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperSimpleApp
>
> # The following property instructs the Java Service Wrapper
> # to append the system PATH to the java.library.path at
> # runtime.  This is used in order to load secondary DLLs.
> wrapper.java.library.path.append_system_path=true
>
> # Set the debug flag.
> wrapper.debug=false
>
> # Allow 5 minutes between pings
> wrapper.ping.interval=300
>
> # Do NOT try to restart the JVM
> wrapper.disable_restarts=TRUE
>
> # Number of seconds to wait for the ping to timeout
> wrapper.ping.timeout=900
>
> Our current thought is that we could set the value of the
> wrapper.ping.timeout property to be zero.  This should prevent the Java
> Service Wrapper from attempting to ping the WrapperSimpleApp and ultimately
> prevent the false detection of a hung JVM which was causing the JVM to be
> shut down in the middle of normal processing.  Does that sound about
> accurate?
>
> Has anyone else out there experienced this or a similar problem?  If so,
> what was your solution?  Is there an alternate property that I should be
> focused on instead of the wrapper.ping.timeout property?  I am aware of the
> wrapper.cpu.timeout property, but I have not seen any messages in the log to
> indicate that the wrapper has not received any CPU time for N seconds.
>
> I would appreciate any input that you may have to offer.
>
> Patrick M. Farrell, PMP, CSM
> Consultant Mentor
> Quick Solutions, Inc.
> A four time INC 500 winner!
> pfa...@qu...
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great
> prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Wrapper-user mailing list
> Wra...@li...
> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>
>