Re: [Wrapper-user] JVM appears hung: VMware, verbose garbage collection

[Nathan W. <wr...@de...>]

Thanks Leif, we're running the ping timeout now at a higher value (360).  We
dropped it to 10 trying to force the wrapper to kill the process, trying to
figure out why it was never doing a clean shutdown.
Now that it is clear the shutdown will always be a hard kill we can stop
that testing.

We're still stuck with a server that cannot be started after a hard kill
though.  After a hard kill, the server process starts but the application
configuration files, etc are all missing, and it just fails along
uselessly.  When we stop the server and clean up the temp folders it's all
fine, and when the server is started and stopped normally it's fine.  But we
cannot seem to survive a hard shutdown.  We're looking at ways to add a
batch script to be called before the server is started, so we can clean up
the temp folder.  One idea that seems promising is to replace the
wrapper.java.command with a batch file that first cleans the temp folder and
then calls java.  Do you have any other suggestions for hooking into
application startup?

The batch file would be similar to this:

IF EXIST "%JBOSS_HOME%\server\default\tmp" (
  rmdir /S /Q "%JBOSS_HOME%\server\default\tmp"
)

"%JAVA_HOME%\bin\java" %*

Thanks again for your help Leif
Nathan


On 10/29/07, Leif Mortenson <le...@ta...> wrote:
>
> Nathan,
> Java does indeed appear frozen if its memory gets swapped to disk
> or during long GC cycles.    To an outside process, the JVM is indeed
> frozen while either event happens.
>
> Looking at the wrapper.conf that you posted, you have your
> wrapper.ping.timeout set to 10 seconds.  Why did you set it to a
> value shorter than the default of 30 seconds?  My first suggestion
> would be to set the timeout to 60 or 120 seconds and see how that
> works for you.
>
> An old version of the Wrapper started to shutdown cleanly when
> a ping timeout was encountered.  But if the JVM is really frozen,
> the shutdown would of course still fail.   If the JVM was not really
> frozen then the JVM would still either be shutdown or killed.  If
> the "frozen" state is an artifact of GC or of memory swapping, then
> a better solution is to increase the ping timeout.
>
> I have used the Wrapper under VMWare quite a bit.  But not in
> cases where the JVM or VM were under heavy loads.  I have seen
> problems when the host OS is under heavy loads, but that is not
> surprising as VMWare becomes pretty doggish under those
> circumstances.
>
> One thing that greatly improved the VMWare performance in cases
> where the host OS was under load was to disable VMWare from
> allowing the VM's memory to be swapped.  Force it to stay in active
> memory.  That gets rid of the long freezes.
>
> Cheers,
> Leif
>
> Nathan Wray wrote:
> >
> > Hi;
> >
> > I have an application being deployed onto JBoss on Windows under
> > VMware.  JBoss is running as a service under wrapper.
> > Although the application has run elsewhere for long periods of time
> > with no issues, we're seeing a problem under wrapper.
> > Once the app has been running for 1-2 hours, we see:
> >
> > ERROR  | wrapper  | 2007/10/26 16:25:17 | JVM appears hung: Timed out
> > waiting for signal from JVM.
> > ERROR  | wrapper  | 2007/10/26 16:25:18 | JVM did not exit on request,
> > terminated
> >
> > Our app is deployed as an EAR file, and if the JBoss server process
> > doesn't shut down cleanly it won't come back up without deleting the
> > tmp, work and data folders.  So once wrapper has gunned the server
> > it's dead until someone manually intervenes.
> >
> > Using the default ticks timer we see messages like this periodically:
> >
> >  The timer fell behind the system clock by 14600ms.
> >
> > Our server is a VMware instance, so it is possible that the VM could
> > be getting swapped out or starved.  This would probably appear to an
> > application as though the clock and timers were jumping out of sync in
> > ways that normally are impossible.
> > Is anyone successfully using wrapper under VMware?
> >
> > Also we're running with verbose garbage collection, which I understand
> > can hang the server momentarily.  Are there known issues with verbose
> > garbage collection under wrapper?
> >
> > Looking through the code it doesn't appear that wrapper tries to shut
> > down the server once it decides there is a hang; it looks like it goes
> > right into WRAPPER_JSTATE_KILLING rather than WRAPPER_JSTATE_STOPPING;
> > so that the shutdownTimeout parameter does not come into play.  Would
> > it be possible to attempt a clean shutdown before killing the server?
> >
> >
> > Selected settings below.  Note right now we're trying to address the
> > hard kill that leaves the server unusable, we will eventually bump up
> > the ping timeout to try and avoid the kill in the first place.
> >
> > # Number of seconds between Wrapper ping requests to the JVM.
> > wrapper.ping.interval=5
> >
> > #Number of seconds to allow between the wrapper pinging the JVM and
> > the response.
> > wrapper.ping.timeout=10
> >
> > #Number of seconds between JVM reports stopped and the JVM actually
> > terminates.
> > wrapper.jvm_exit.timeout=60
> > wrapper.shutdown.timeout=60
> > wrapper.restart.delay=60
> >
> > wrapper.timer_fast_threshold=0
> > wrapper.timer_slow_threshold=10
> >
> > # Force reload of wrapper config on JVM restart
> > wrapper.restart.reload_configuration=TRUE
> >
> > Thanks for any insight you can bring.
> >
> > Nathan
>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems?  Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >> http://get.splunk.com/
> _______________________________________________
> Wrapper-user mailing list
> Wra...@li...
> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>



-- 
Nathan Wray
nw...@de...
--