Menu
▾
▴
Re: [Wrapper-user] JVM appears hung: Timed out waiting for signal from JVM
|
From: Leif M. <le...@ta...> - 2008-07-23 07:45:50
|
Patrick, I agree that it does not appear to be doing you any good. I am wondering why you are needing to play with it it all thought. What is the memory usage of your application vs the physical memory of the machine? Is this being caused by swapping? Does your application modify the thread priority of any of its threads? Are other applications consuming lots of CPU? It may be possible to improve the way the Wrapper operates under these conditions if I can learn exactly what is happening. Thanks, Leif On Wed, Jul 23, 2008 at 3:52 AM, <PFa...@qu...> wrote: > Leif, > > Thank you for your reply. We will schedule the configuration changes as > part of our next release. > > Assuming that we continue to experience the same problem in the future, is > there any reason why we would want to continue messing around with the value > of the property (wrapper.ping.timeout) instead of turning off the > functionality altogether by setting it to zero? I guess I don't see what > benefit it is providing when we are not using the restart functionality. > > Patrick M. Farrell, PMP, CSM > Consultant Mentor > Quick Solutions, Inc. > A four time INC 500 winner! > pfa...@qu... > > -----wra...@li... wrote: ----- > > To: wra...@li... > From: "Leif Mortenson" <le...@ta...> > Sent by: wra...@li... > Date: 07/22/2008 01:27PM > Subject: Re: [Wrapper-user] JVM appears hung: Timed out waiting for signal > from JVM > > Patrick, > If you are having ping timeout problems, it is usually caused by the > application using more memory than the system has available in > Physical Memory. Java can get 100-1000x slower when the OS does even > a little bit of swapping of its memory space. Unless you are changing > thread priorities in your application, the Wrapper will generally not > lose the ability to respond to pings even when the CPU usage is fairly > high. > > Regardless, the property needed to extend the amount of time the > Wrapper will wait for the JVM is as you mentioned: > wrapper.ping.timeout > > You might want to try setting wrapper.debug=true. This will let you > see exactly when pings are send and responded to. > > If you are still getting the timeouts with a ping timeout of 900 then > the JVM must be eating too much serial for more than 15minutes. > > As a test, you might want to try setting the ping timeout of 1800. > > Cheers, > Leif > > On Wed, Jul 23, 2008 at 2:18 AM, <PFa...@qu...> wrote: >> Hello, >> >> The purpose of this post is to determine the most appropriate course of >> action given our current situation. >> >> Background: >> The application is running the 3.3.0 x86 release of the Java Service >> Wrapper >> and using the WrapperSimpleApp main class at startup. The supporting >> hardware has 8 CPUs running 4 instances of the application. The >> processing >> for this service is quite CPU and memory intensive. It is not uncommon >> for >> the CPU utilization on the box to reach 100% for 3-5 minutes at a time, >> and >> if four large files arrived in quick succession, the CPU utilization could >> potentially be pegged for as long as 15 minutes. >> >> Symptom: >> The Java Service Wrapper is reporting that the JVM appears to be hung and >> that the timeout expired while waiting on a signal from the JVM. The Java >> Service Wrapper then shuts down the JVM even though we believe the >> application is processing normally. Note: We have explicitly disabled >> the >> automatic restart functionality as we would like our existing monitoring >> software to detect and report the problem with the instance of the >> application. The following snippet is from the wrapper.log file: >> >> ERROR | wrapper | 2008/07/21 16:15:53 | JVM appears hung: Timed out >> waiting for signal from JVM. >> STATUS | wrapper | 2008/07/21 16:15:54 | JVM Restarts disabled. Shutting >> down. >> STATUS | wrapper | 2008/07/21 16:15:54 | <-- Wrapper Stopped >> >> The wrapper configuration is as follows: >> >> # ----------------------------------------------------------------- >> # This file contains the configuration for the Java Service Wrapper >> # ----------------------------------------------------------------- >> >> # Set the name of the service. >> wrapper.name=VPC Gen2 Capture Service 1 >> >> # Set the location of the wrapper log >> wrapper.logfile=../logs/wrapper.1.log >> >> # Set the java command line options. >> wrapper.java.additional.1=-DFILE_PREFIX=//localhost/data >> wrapper.java.additional.2=-Dlog4j.configuration=log4j.1.xml >> >> wrapper.java.additional.3=-DCAPTURE_SERVICES_LOG_DIR=D:\vpc\capture-services\ >> >> wrapper.java.additional.4=-Dcom.sun.management.jmxremote.authenticate=false >> wrapper.java.additional.5=-Dcom.sun.management.jmxremote.port=1151 >> wrapper.java.additional.6=-Dcom.sun.management.jmxremote.ssl=false >> wrapper.java.additional.7=-Xmx1150m >> wrapper.java.additional.8=-Xms512m >> wrapper.java.additional.9=-XX:MaxPermSize=192m >> wrapper.java.additional.10=-server >> >> # Set the application classpath. >> # The application classpath must contain wrapper.jar! >> wrapper.java.classpath.1=../lib/wrapper.jar >> wrapper.java.classpath.2=../../lib/vpc-framework-1.1.3.jar >> wrapper.java.classpath.3=../../lib/capture-services-1.1.3.jar >> >> # ----------------------------------------------------------------- >> # The following properties should not need to be modified. >> # ----------------------------------------------------------------- >> >> # Set the application classname and optionally any >> # command-line parameters that must be passed. >> wrapper.app.parameter.1=com.jpmc.vpc.core.StartApplication >> >> # Set the Java Library Path. >> wrapper.java.library.path.1=../lib >> >> # Set the Java command. >> wrapper.java.command=%JAVA_HOME%/bin/java >> >> # Set the Java Service Wrapper main class. >> wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperSimpleApp >> >> # The following property instructs the Java Service Wrapper >> # to append the system PATH to the java.library.path at >> # runtime. This is used in order to load secondary DLLs. >> wrapper.java.library.path.append_system_path=true >> >> # Set the debug flag. >> wrapper.debug=false >> >> # Allow 5 minutes between pings >> wrapper.ping.interval=300 >> >> # Do NOT try to restart the JVM >> wrapper.disable_restarts=TRUE >> >> # Number of seconds to wait for the ping to timeout >> wrapper.ping.timeout=900 >> >> Our current thought is that we could set the value of the >> wrapper.ping.timeout property to be zero. This should prevent the Java >> Service Wrapper from attempting to ping the WrapperSimpleApp and >> ultimately >> prevent the false detection of a hung JVM which was causing the JVM to be >> shut down in the middle of normal processing. Does that sound about >> accurate? >> >> Has anyone else out there experienced this or a similar problem? If so, >> what was your solution? Is there an alternate property that I should be >> focused on instead of the wrapper.ping.timeout property? I am aware of >> the >> wrapper.cpu.timeout property, but I have not seen any messages in the log >> to >> indicate that the wrapper has not received any CPU time for N seconds. >> >> I would appreciate any input that you may have to offer. >> >> Patrick M. Farrell, PMP, CSM >> Consultant Mentor >> Quick Solutions, Inc. >> A four time INC 500 winner! >> pfa...@qu... >> >> ------------------------------------------------------------------------- >> This SF.Net email is sponsored by the Moblin Your Move Developer's >> challenge >> Build the coolest Linux based applications with Moblin SDK & win great >> prizes >> Grand prize is a trip for two to an Open Source event anywhere in the >> world >> http://moblin-contest.org/redirect.php?banner_id=100&url=/ >> _______________________________________________ >> Wrapper-user mailing list >> Wra...@li... >> https://lists.sourceforge.net/lists/listinfo/wrapper-user >> >> > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great > prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Wrapper-user mailing list > Wra...@li... > https://lists.sourceforge.net/lists/listinfo/wrapper-user > > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great > prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Wrapper-user mailing list > Wra...@li... > https://lists.sourceforge.net/lists/listinfo/wrapper-user > > |
