Re: [Wrapper-user] JVM appears hung: Timed out waiting for signalfrom JVM. problem

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Simon,
Thanks for the log. I need to confirm as it makes a big difference
how I look into this. Is what you sent the very END of your log file
or does it continue. You had mentioned that the Wrapper process
was hanging and not restarting the JVM. I want to verify, A) that the
Wrapper process is hanging, and B) that the log you posted is indeed
the last output before it does so.
Also, from your configuration file properties, it looks like you are
using version 3.2.3. Is that correct?

Cheers,
Leif

Simon Chan wrote:
> Hi Leif,
> The following is the error after add the value that u mentioned before:
> STATUS | wrapper | 2007/06/27 01:13:45 | Ticks=0008ce46,
> WrapperState=STARTED, JVMState=STARTED JVMStateTimeoutTicks=0008ce55
> (1s), Exit=false, Restart=false
> STATUS | wrapper | 2007/06/27 01:13:45 | Loop: handle wrapper state:
> STARTED
> STATUS | wrapper | 2007/06/27 01:13:46 | Loop: handle jvm state: STARTED
> STATUS | wrapper | 2007/06/27 01:13:46 | Loop: sleep
> STATUS | wrapper | 2007/06/27 01:13:46 | Loop: maintain logger
> STATUS | wrapper | 2007/06/27 01:13:46 | Loop: process jvm output
> STATUS | wrapper | 2007/06/27 01:13:46 | Loop: process socket
> STATUS | wrapper | 2007/06/27 01:13:47 | Loop: maintain logger(2)
> STATUS | wrapper | 2007/06/27 01:13:47 | Ticks=0008ce52,
> WrapperState=STARTED, JVMState=STARTED JVMStateTimeoutTicks=0008ce55
> (0s), Exit=false, Restart=false
> STATUS | wrapper | 2007/06/27 01:13:47 | Loop: handle wrapper state:
> STARTED
> STATUS | wrapper | 2007/06/27 01:13:47 | Loop: handle jvm state: STARTED
> *ERROR | wrapper | 2007/06/27 01:13:47 | JVM appears hung: Timed out
> waiting for signal from JVM.
> STATUS | wrapper | 2007/06/27 01:13:47 | Set Java State STARTED -> KILLING
> STATUS | wrapper | 2007/06/27 01:13:47 | Set Java State KILLING
> Timeout 0008ce55 -> 0008ce52
> STATUS | wrapper | 2007/06/27 01:13:47 | Loop: sleep
> STATUS | wrapper | 2007/06/27 01:13:48 | Loop: maintain logger
> STATUS | wrapper | 2007/06/27 01:13:48 | Loop: process jvm output
> STATUS | wrapper | 2007/06/27 01:13:48 | Loop: process socket
> STATUS | wrapper | 2007/06/27 01:13:48 | Loop: maintain logger(2)*
> **
> do u have any idea on it?
> For server, our server has 2G RAM and running mod_jk + tomcat. we are
> running 2 x tomcat now.
> Regards
> Simon.
> ----- Original Message -----
> From: "Leif Mortenson" <le...@ta...
> <mailto:le...@ta...>>
> To: <wra...@li...
> <mailto:wra...@li...>>
> Sent: Tuesday, June 26, 2007 2:09 PM
> Subject: Re: [Wrapper-user] JVM appears hung: Timed out waiting for
> signalfrom JVM. problem
>
> > Simon,
> > Your configuration is forcing the JVM to allocate 768MB of memory
> > on startup? How much memory does your system have available?
> > Are you sure the JVM is able to run without any disk swapping taking
> > place? Java gets VERY slow when its memory is being swapped to disk.
> > We are talking about 100-1000x slower. This can cause the JVM to
> > appear to be frozen for long periods of time.
> >
> > The log looks like the JVM is freezing up for some reason. The
> > Wrapper detects that and then should be attempting a restart.
> > Extending the ping timeout would make the wrapper take longer to
> > decide that the JVM is frozen, but it will not solve the problem if it
> > really is frozen.
> >
> > Is what you posted the end of your wrapper.log file? I would
> > expect more output after that point. If that is the end of your
> > log file, try adding the following property. It greatly increases the
> > output but will show whether or not the main loop of the wrapper
> > is frozen.
> >
> > wrapper.state_output=TRUE
> >
> > If that freezes, then adding the following as well will provide
> > even more information about the precise location of the freeze.
> >
> > wrapper.loop_output=TRUE
> >
> > Cheers,
> > Leif
> >
> > Simon Chan wrote:
> >> Hi leif,
> >> have any news on it?
> >> I tried to change wrapper.ping.timeout to 180 and seem still hang.
> >> is it CPU 100% fully use or no memory?
> >> Regards
> >> Simon.
> >>
> >> ----- Original Message -----
> >> *From:* Simon Chan (Funmobile) <mailto:sim...@fu...>
> >> *To:* wra...@li...
> <mailto:wra...@li...>
> >> <mailto:wra...@li...>
> >> *Sent:* Sunday, June 24, 2007 9:55 AM
> >> *Subject:* Re: [Wrapper-user] JVM appears hung: Timed out waiting
> >> for signalfrom JVM. problem
> >>
> >> Hi Leif,
> >> the following is the wrapper.conf:
> >> #********************************************************************
> >> # TestWrapper Properties
> >> #
> >> # NOTE - Please use src/conf/wrapper.conf.in as a template for your
> >> # own application rather than the values used for the
> >> # TestWrapper sample.
> >> #********************************************************************
> >> # Java Application
> >> wrapper.java.command=/usr/java/jdk1.5.0_07/bin/java
> >> # Java Main class. This class must implement the WrapperListener
> >> interface
> >> # or guarantee that the WrapperManager class is initialized. Helper
> >> # classes are provided to do this for you. See the Integration section
> >> # of the documentation for details.
> >> #wrapper.java.mainclass=org.tanukisoftware.wrapper.test.Main
> >> wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperStartStopApp
> >> # Java Classpath (include wrapper.jar) Add class path elements as
> >> # needed starting from 1
> >> # wrapper.java.classpath.1=../lib/wrappertest.jar
> >> # wrapper.java.classpath.2=../lib/wrapper.jar
> >>
> wrapper.java.classpath.1=/usr/local/wrapper-linux-x86-32-3.2.3/lib/wrapper.jar
> >> wrapper.java.classpath.2=/usr/apache-tomcat-5.5.17/bin/bootstrap.jar
> >>
> wrapper.java.classpath.3=/usr/apache-tomcat-5.5.17/bin/commons-logging-api.jar
> >> wrapper.java.classpath.4=/usr/java/jdk1.5.0_07/lib/tools.jar
> >> # Java Library Path (location of Wrapper.DLL or libwrapper.so)
> >> # wrapper.java.library.path.1=../lib
> >> wrapper.java.library.path.1=/usr/local/wrapper-linux-x86-32-3.2.3/lib
> >> # Java Additional Parameters
> >> #wrapper.java.additional.1=
> >>
> wrapper.java.additional.1=-Djava.endorsed.dirs=/usr/apache-tomcat-5.5.17/common/endorsed
> >> wrapper.java.additional.2=-Dcatalina.base=/usr/apache-tomcat-5.5.17
> >> wrapper.java.additional.3=-Dcatalina.home=/usr/apache-tomcat-5.5.17
> >>
> wrapper.java.additional.4=-Djava.io.tmpdir=/usr/apache-tomcat-5.5.17/temp
> >> wrapper.java.additional.5=-XX:MaxPermSize=64m
> >> # Initial Java Heap Size (in MB)
> >> #wrapper.java.initmemory=3
> >> wrapper.java.initmemory=768
> >> # Maximum Java Heap Size (in MB)
> >> #wrapper.java.maxmemory=64
> >> wrapper.java.maxmemory=768
> >> # Application parameters. Add parameters as needed starting from 1
> >> #wrapper.app.parameter.1=
> >> wrapper.app.parameter.1=org.apache.catalina.startup.Bootstrap
> >> wrapper.app.parameter.2=1
> >> wrapper.app.parameter.3=start
> >> wrapper.app.parameter.4=org.apache.catalina.startup.Bootstrap
> >> wrapper.app.parameter.5=true
> >> wrapper.app.parameter.6=1
> >> wrapper.app.parameter.7=stop
> >> #********************************************************************
> >> # Wrapper Logging Properties
> >> #********************************************************************
> >> # Format of output for the console. (See docs for formats)
> >> wrapper.console.format=PM
> >> # Log Level for console output. (See docs for log levels)
> >> wrapper.console.loglevel=INFO
> >> # Log file to use for wrapper output logging.
> >> wrapper.logfile=/var/log/tomcat/tomcat-1.log
> >> # Format of output for the log file. (See docs for formats)
> >> wrapper.logfile.format=LPTM
> >> # Log Level for log file output. (See docs for log levels)
> >> wrapper.logfile.loglevel=DEBUG
> >> # Maximum size that the log file will be allowed to grow to before
> >> # the log is rolled. Size is specified in bytes. The default value
> >> # of 0, disables log rolling. May abbreviate with the 'k' (kb) or
> >> # 'm' (mb) suffix. For example: 10m = 10 megabytes.
> >> wrapper.logfile.maxsize=0
> >> # Maximum number of rolled log files which will be allowed before old
> >> # files are deleted. The default value of 0 implies no limit.
> >> wrapper.logfile.maxfiles=0
> >> # Log Level for sys/event log output. (See docs for log levels)
> >> wrapper.syslog.loglevel=DEBUG
> >> wrapper.ping.timeout=60
> >> wrapper.statusfile=/var/run/tomcat/wrapper_1.status
> >> wrapper.filter.trigger.1=java.lang.OutOfMemoryError
> >> wrapper.filter.action.1=RESTART
> >> and we use it for monitor tomcat 5.5.17. and the following is the
> >> debug log:
> >> DEBUG | wrapperp | 2007/06/23 16:47:17 | read a packet PING : ok
> >> DEBUG | wrapper | 2007/06/23 16:47:17 | Got ping response from JVM
> >> DEBUG | wrapperp | 2007/06/23 16:47:22 | send a packet PING : ping
> >> INFO | jvm 2 | 2007/06/23 16:47:22 | Received a packet PING : ping
> >> INFO | jvm 2 | 2007/06/23 16:47:22 | Send a packet PING : ok
> >> DEBUG | wrapperp | 2007/06/23 16:47:22 | read a packet PING : ok
> >> DEBUG | wrapper | 2007/06/23 16:47:22 | Got ping response from JVM
> >> DEBUG | wrapperp | 2007/06/23 16:47:26 | send a packet PING : ping
> >> DEBUG | wrapperp | 2007/06/23 16:47:31 | send a packet PING : ping
> >> DEBUG | wrapperp | 2007/06/23 16:47:35 | send a packet PING : ping
> >> DEBUG | wrapperp | 2007/06/23 16:47:40 | send a packet PING : ping
> >> DEBUG | wrapperp | 2007/06/23 16:47:44 | send a packet PING : ping
> >> DEBUG | wrapperp | 2007/06/23 16:47:49 | send a packet PING : ping
> >> DEBUG | wrapperp | 2007/06/23 16:47:53 | send a packet PING : ping
> >> DEBUG | wrapperp | 2007/06/23 16:47:58 | send a packet PING : ping
> >> DEBUG | wrapperp | 2007/06/23 16:48:02 | send a packet PING : ping
> >> DEBUG | wrapperp | 2007/06/23 16:48:07 | send a packet PING : ping
> >> DEBUG | wrapperp | 2007/06/23 16:48:11 | send a packet PING : ping
> >> DEBUG | wrapperp | 2007/06/23 16:48:16 | send a packet PING : ping
> >> DEBUG | wrapperp | 2007/06/23 16:48:20 | send a packet PING : ping
> >> DEBUG | wrapperp | 2007/06/23 16:48:25 | send a packet PING : ping
> >> DEBUG | wrapperp | 2007/06/23 16:48:29 | send a packet PING : ping
> >> *ERROR | wrapper | 2007/06/23 16:48:32 | JVM appears hung: Timed
> >> out waiting for signal from JVM.*
> >> **
> >> wrapper seeems hang after this and cannot try to restart JVM. it
> >> is so strange?
> >> I need to try to increase the value of ping timeout?
> >> Regards
> >> Simon.
> >> ----- Original Message -----
> >> From: "Leif Mortenson" <le...@ta...
> <mailto:le...@ta...>
> >> <mailto:le...@ta...>>
> >> To: <wra...@li...
> <mailto:wra...@li...>
> >> <mailto:wra...@li...>>
> >> Sent: Sunday, June 24, 2007 8:45 AM
> >> Subject: Re: [Wrapper-user] JVM appears hung: Timed out waiting
> >> for signal from JVM. problem
> >>
> >> > Simon,
> >> > Can you post your wrapper.conf file? Also if you are able to
> >> > reproduce this, can you set the wrapper.debug=true property
> >> > and post the resulting wrapper.conf file as well?
> >> >
> >> > >From what you posted, it looks like the Wrapper is detecting
> >> > that the JVM is hung, at which point it should be killing and
> >> > restarting the JVM process. I am not clear exactly what you
> >> > are see at that point however.
> >> >
> >> > Cheers,
> >> > Leif
> >> >
> >> > Simon Chan (Funmobile) wrote:
> >> > > hi all,
> >> > > I am using wrapper to monitor my tomcat 5.5.17 (use java 1.5) now.
> >> > > but I see that it is hung and show the following error:
> >> > > INFO | jvm 1 | 2007/06/22 18:34:25 | Send a packet PING : ok
> >> > > ERROR | wrapper | 2007/06/22 18:35:36 | JVM appears hung:
> >> Timed out
> >> > > waiting for signal from JVM.
> >> > > wrapper seems don't try to restart it and just leave it and make
> >> > > tomcat die.
> >> > > do u have any idea?
> >> > > Regards
> >> > > Simon.
> >> > >
> >>
> ------------------------------------------------------------------------
> >> > >
> >> > >
> >>
> -------------------------------------------------------------------------
> >> > > This SF.net email is sponsored by DB2 Express
> >> > > Download DB2 Express C - the FREE version of DB2 express and take
> >> > > control of your XML. No limits. Just data. Click to get it now.
> >> > > http://sourceforge.net/powerbar/db2/
> >> > >
> >>
> ------------------------------------------------------------------------
> >> > >
> >> > > _______________________________________________
> >> > > Wrapper-user mailing list
> >> > > Wra...@li...
> <mailto:Wra...@li...>
> >> <mailto:Wra...@li...>
> >> > > https://lists.sourceforge.net/lists/listinfo/wrapper-user
> >> > >
> >> >
> >> >
> >> >
> >>
> ------------------------------------------------------------------------
> >> >
> >>
> -------------------------------------------------------------------------
> >> > This SF.net email is sponsored by DB2 Express
> >> > Download DB2 Express C - the FREE version of DB2 express and take
> >> > control of your XML. No limits. Just data. Click to get it now.
> >> > http://sourceforge.net/powerbar/db2/
> >>
> ------------------------------------------------------------------------
> >> > _______________________________________________
> >> > Wrapper-user mailing list
> >> > Wra...@li...
> <mailto:Wra...@li...>
> >> <mailto:Wra...@li...>
> >> > https://lists.sourceforge.net/lists/listinfo/wrapper-user
> >> >
> >>
> ------------------------------------------------------------------------
> >> No virus found in this incoming message.
> >> Checked by AVG Free Edition.
> >> Version: 7.5.472 / Virus Database: 269.9.6/862 - Release Date:
> >> 6/22/2007 3:04 PM
> >>
> >>
> ------------------------------------------------------------------------
> >>
> -------------------------------------------------------------------------
> >> This SF.net email is sponsored by DB2 Express
> >> Download DB2 Express C - the FREE version of DB2 express and take
> >> control of your XML. No limits. Just data. Click to get it now.
> >> http://sourceforge.net/powerbar/db2/
> >>
> >>
> ------------------------------------------------------------------------
> >> _______________________________________________
> >> Wrapper-user mailing list
> >> Wra...@li...
> <mailto:Wra...@li...>
> >> https://lists.sourceforge.net/lists/listinfo/wrapper-user
> >>
> >>
> ------------------------------------------------------------------------
> >>
> >> Anti-virus and anti-spam by FortiGate
> >>
> >>
> ------------------------------------------------------------------------
> >>
> >>
> -------------------------------------------------------------------------
> >> This SF.net email is sponsored by DB2 Express
> >> Download DB2 Express C - the FREE version of DB2 express and take
> >> control of your XML. No limits. Just data. Click to get it now.
> >> http://sourceforge.net/powerbar/db2/
> >>
> ------------------------------------------------------------------------
> >>
> >> _______________________________________________
> >> Wrapper-user mailing list
> >> Wra...@li...
> <mailto:Wra...@li...>
> >> https://lists.sourceforge.net/lists/listinfo/wrapper-user
> >>
> >
> >
> >
> ------------------------------------------------------------------------
> >
> -------------------------------------------------------------------------
> > This SF.net email is sponsored by DB2 Express
> > Download DB2 Express C - the FREE version of DB2 express and take
> > control of your XML. No limits. Just data. Click to get it now.
> > http://sourceforge.net/powerbar/db2/
> ------------------------------------------------------------------------
> > _______________________________________________
> > Wrapper-user mailing list
> > Wra...@li...
> <mailto:Wra...@li...>
> > https://lists.sourceforge.net/lists/listinfo/wrapper-user
> >
> ------------------------------------------------------------------------
>
> Anti-virus and anti-spam by FortiGate
>   
> ------------------------------------------------------------------------
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> ------------------------------------------------------------------------
>
> _______________________________________________
> Wrapper-user mailing list
> Wra...@li...
> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>   