wrapper-user

[Wrapper-user] JVM appears hung: VMware, verbose garbage collection

[Nathan W. <wr...@de...>]

Hi;

I have an application being deployed onto JBoss on Windows under VMware.
JBoss is running as a service under wrapper.
Although the application has run elsewhere for long periods of time with no
issues, we're seeing a problem under wrapper.
Once the app has been running for 1-2 hours, we see:

ERROR  | wrapper  | 2007/10/26 16:25:17 | JVM appears hung: Timed out
waiting for signal from JVM.
ERROR  | wrapper  | 2007/10/26 16:25:18 | JVM did not exit on request,
terminated

Our app is deployed as an EAR file, and if the JBoss server process doesn't
shut down cleanly it won't come back up without deleting the tmp, work and
data folders.  So once wrapper has gunned the server it's dead until someone
manually intervenes.

Using the default ticks timer we see messages like this periodically:

 The timer fell behind the system clock by 14600ms.

Our server is a VMware instance, so it is possible that the VM could be
getting swapped out or starved.  This would probably appear to an
application as though the clock and timers were jumping out of sync in ways
that normally are impossible.
Is anyone successfully using wrapper under VMware?

Also we're running with verbose garbage collection, which I understand can
hang the server momentarily.  Are there known issues with verbose garbage
collection under wrapper?

Looking through the code it doesn't appear that wrapper tries to shut down
the server once it decides there is a hang; it looks like it goes right into
WRAPPER_JSTATE_KILLING rather than WRAPPER_JSTATE_STOPPING; so that the
shutdownTimeout parameter does not come into play.  Would it be possible to
attempt a clean shutdown before killing the server?


Selected settings below.  Note right now we're trying to address the hard
kill that leaves the server unusable, we will eventually bump up the ping
timeout to try and avoid the kill in the first place.

# Number of seconds between Wrapper ping requests to the JVM.
wrapper.ping.interval=5

#Number of seconds to allow between the wrapper pinging the JVM and the
response.
wrapper.ping.timeout=10

#Number of seconds between JVM reports stopped and the JVM actually
terminates.
wrapper.jvm_exit.timeout=60
wrapper.shutdown.timeout=60
wrapper.restart.delay=60

wrapper.timer_fast_threshold=0
wrapper.timer_slow_threshold=10

# Force reload of wrapper config on JVM restart
wrapper.restart.reload_configuration=TRUE

Thanks for any insight you can bring.

Nathan

-- 
Nathan Wray

Re: [Wrapper-user] JVM appears hung: VMware, verbose garbage collection

[Leif M. <le...@ta...>]

Nathan,
Java does indeed appear frozen if its memory gets swapped to disk
or during long GC cycles.    To an outside process, the JVM is indeed
frozen while either event happens.

Looking at the wrapper.conf that you posted, you have your
wrapper.ping.timeout set to 10 seconds.  Why did you set it to a
value shorter than the default of 30 seconds?  My first suggestion
would be to set the timeout to 60 or 120 seconds and see how that
works for you.

An old version of the Wrapper started to shutdown cleanly when
a ping timeout was encountered.  But if the JVM is really frozen,
the shutdown would of course still fail.   If the JVM was not really
frozen then the JVM would still either be shutdown or killed.  If
the "frozen" state is an artifact of GC or of memory swapping, then
a better solution is to increase the ping timeout.

I have used the Wrapper under VMWare quite a bit.  But not in
cases where the JVM or VM were under heavy loads.  I have seen
problems when the host OS is under heavy loads, but that is not
surprising as VMWare becomes pretty doggish under those
circumstances.

One thing that greatly improved the VMWare performance in cases
where the host OS was under load was to disable VMWare from
allowing the VM's memory to be swapped.  Force it to stay in active
memory.  That gets rid of the long freezes.

Cheers,
Leif

Nathan Wray wrote:
>
> Hi;
>
> I have an application being deployed onto JBoss on Windows under 
> VMware.  JBoss is running as a service under wrapper.
> Although the application has run elsewhere for long periods of time 
> with no issues, we're seeing a problem under wrapper.
> Once the app has been running for 1-2 hours, we see:
>
> ERROR  | wrapper  | 2007/10/26 16:25:17 | JVM appears hung: Timed out 
> waiting for signal from JVM.
> ERROR  | wrapper  | 2007/10/26 16:25:18 | JVM did not exit on request, 
> terminated
>
> Our app is deployed as an EAR file, and if the JBoss server process 
> doesn't shut down cleanly it won't come back up without deleting the 
> tmp, work and data folders.  So once wrapper has gunned the server 
> it's dead until someone manually intervenes.
>
> Using the default ticks timer we see messages like this periodically:
>
>  The timer fell behind the system clock by 14600ms.
>
> Our server is a VMware instance, so it is possible that the VM could 
> be getting swapped out or starved.  This would probably appear to an 
> application as though the clock and timers were jumping out of sync in 
> ways that normally are impossible.
> Is anyone successfully using wrapper under VMware?
>
> Also we're running with verbose garbage collection, which I understand 
> can hang the server momentarily.  Are there known issues with verbose 
> garbage collection under wrapper?
>
> Looking through the code it doesn't appear that wrapper tries to shut 
> down the server once it decides there is a hang; it looks like it goes 
> right into WRAPPER_JSTATE_KILLING rather than WRAPPER_JSTATE_STOPPING; 
> so that the shutdownTimeout parameter does not come into play.  Would 
> it be possible to attempt a clean shutdown before killing the server?
>
>
> Selected settings below.  Note right now we're trying to address the 
> hard kill that leaves the server unusable, we will eventually bump up 
> the ping timeout to try and avoid the kill in the first place.
>
> # Number of seconds between Wrapper ping requests to the JVM.
> wrapper.ping.interval=5
>
> #Number of seconds to allow between the wrapper pinging the JVM and 
> the response.
> wrapper.ping.timeout=10
>
> #Number of seconds between JVM reports stopped and the JVM actually 
> terminates.
> wrapper.jvm_exit.timeout=60
> wrapper.shutdown.timeout=60
> wrapper.restart.delay=60
>
> wrapper.timer_fast_threshold=0
> wrapper.timer_slow_threshold=10
>
> # Force reload of wrapper config on JVM restart
> wrapper.restart.reload_configuration=TRUE
>
> Thanks for any insight you can bring.
>
> Nathan

Re: [Wrapper-user] JVM appears hung: VMware, verbose garbage collection

[Nathan W. <wr...@de...>]

Thanks Leif, we're running the ping timeout now at a higher value (360).  We
dropped it to 10 trying to force the wrapper to kill the process, trying to
figure out why it was never doing a clean shutdown.
Now that it is clear the shutdown will always be a hard kill we can stop
that testing.

We're still stuck with a server that cannot be started after a hard kill
though.  After a hard kill, the server process starts but the application
configuration files, etc are all missing, and it just fails along
uselessly.  When we stop the server and clean up the temp folders it's all
fine, and when the server is started and stopped normally it's fine.  But we
cannot seem to survive a hard shutdown.  We're looking at ways to add a
batch script to be called before the server is started, so we can clean up
the temp folder.  One idea that seems promising is to replace the
wrapper.java.command with a batch file that first cleans the temp folder and
then calls java.  Do you have any other suggestions for hooking into
application startup?

The batch file would be similar to this:

IF EXIST "%JBOSS_HOME%\server\default\tmp" (
  rmdir /S /Q "%JBOSS_HOME%\server\default\tmp"
)

"%JAVA_HOME%\bin\java" %*

Thanks again for your help Leif
Nathan


On 10/29/07, Leif Mortenson <le...@ta...> wrote:
>
> Nathan,
> Java does indeed appear frozen if its memory gets swapped to disk
> or during long GC cycles.    To an outside process, the JVM is indeed
> frozen while either event happens.
>
> Looking at the wrapper.conf that you posted, you have your
> wrapper.ping.timeout set to 10 seconds.  Why did you set it to a
> value shorter than the default of 30 seconds?  My first suggestion
> would be to set the timeout to 60 or 120 seconds and see how that
> works for you.
>
> An old version of the Wrapper started to shutdown cleanly when
> a ping timeout was encountered.  But if the JVM is really frozen,
> the shutdown would of course still fail.   If the JVM was not really
> frozen then the JVM would still either be shutdown or killed.  If
> the "frozen" state is an artifact of GC or of memory swapping, then
> a better solution is to increase the ping timeout.
>
> I have used the Wrapper under VMWare quite a bit.  But not in
> cases where the JVM or VM were under heavy loads.  I have seen
> problems when the host OS is under heavy loads, but that is not
> surprising as VMWare becomes pretty doggish under those
> circumstances.
>
> One thing that greatly improved the VMWare performance in cases
> where the host OS was under load was to disable VMWare from
> allowing the VM's memory to be swapped.  Force it to stay in active
> memory.  That gets rid of the long freezes.
>
> Cheers,
> Leif
>
> Nathan Wray wrote:
> >
> > Hi;
> >
> > I have an application being deployed onto JBoss on Windows under
> > VMware.  JBoss is running as a service under wrapper.
> > Although the application has run elsewhere for long periods of time
> > with no issues, we're seeing a problem under wrapper.
> > Once the app has been running for 1-2 hours, we see:
> >
> > ERROR  | wrapper  | 2007/10/26 16:25:17 | JVM appears hung: Timed out
> > waiting for signal from JVM.
> > ERROR  | wrapper  | 2007/10/26 16:25:18 | JVM did not exit on request,
> > terminated
> >
> > Our app is deployed as an EAR file, and if the JBoss server process
> > doesn't shut down cleanly it won't come back up without deleting the
> > tmp, work and data folders.  So once wrapper has gunned the server
> > it's dead until someone manually intervenes.
> >
> > Using the default ticks timer we see messages like this periodically:
> >
> >  The timer fell behind the system clock by 14600ms.
> >
> > Our server is a VMware instance, so it is possible that the VM could
> > be getting swapped out or starved.  This would probably appear to an
> > application as though the clock and timers were jumping out of sync in
> > ways that normally are impossible.
> > Is anyone successfully using wrapper under VMware?
> >
> > Also we're running with verbose garbage collection, which I understand
> > can hang the server momentarily.  Are there known issues with verbose
> > garbage collection under wrapper?
> >
> > Looking through the code it doesn't appear that wrapper tries to shut
> > down the server once it decides there is a hang; it looks like it goes
> > right into WRAPPER_JSTATE_KILLING rather than WRAPPER_JSTATE_STOPPING;
> > so that the shutdownTimeout parameter does not come into play.  Would
> > it be possible to attempt a clean shutdown before killing the server?
> >
> >
> > Selected settings below.  Note right now we're trying to address the
> > hard kill that leaves the server unusable, we will eventually bump up
> > the ping timeout to try and avoid the kill in the first place.
> >
> > # Number of seconds between Wrapper ping requests to the JVM.
> > wrapper.ping.interval=5
> >
> > #Number of seconds to allow between the wrapper pinging the JVM and
> > the response.
> > wrapper.ping.timeout=10
> >
> > #Number of seconds between JVM reports stopped and the JVM actually
> > terminates.
> > wrapper.jvm_exit.timeout=60
> > wrapper.shutdown.timeout=60
> > wrapper.restart.delay=60
> >
> > wrapper.timer_fast_threshold=0
> > wrapper.timer_slow_threshold=10
> >
> > # Force reload of wrapper config on JVM restart
> > wrapper.restart.reload_configuration=TRUE
> >
> > Thanks for any insight you can bring.
> >
> > Nathan
>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems?  Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >> http://get.splunk.com/
> _______________________________________________
> Wrapper-user mailing list
> Wra...@li...
> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>



-- 
Nathan Wray
nw...@de...
--

Re: [Wrapper-user] JVM appears hung: VMware, verbose garbage collection

[Leif M. <le...@ta...>]

Nathan,
The batch file idea is unfortunately not going to work.  The Wrapper
expects that a JVM will be launched.   It doesn't help now, but I have
been working on the ability to launch external processes in response
to various Wrapper and JVM lifecycle events.

For now, I would suggest changing the main class that you are
passing to the WrapperSimpleApp helper class with a simple
class that does the clean up work for you.  The last line of that
class's main method would simply call the main method of your
real application.

This will let you do the necessary cleanup before the actual
application attempts to initialize itself.

public class StartupHelper
{
    private static void doMyCleanup()
    {
        // ...
    }

    public static void main( String args[] )
    {
        doMyCleanup();

        RealMainClass.main( args );
    }
}

Cheers,
Leif

Nathan Wray wrote:
> Thanks Leif, we're running the ping timeout now at a higher value 
> (360).  We dropped it to 10 trying to force the wrapper to kill the 
> process, trying to figure out why it was never doing a clean shutdown. 
> Now that it is clear the shutdown will always be a hard kill we can 
> stop that testing. 
>
> We're still stuck with a server that cannot be started after a hard 
> kill though.  After a hard kill, the server process starts but the 
> application configuration files, etc are all missing, and it just 
> fails along uselessly.  When we stop the server and clean up the temp 
> folders it's all fine, and when the server is started and stopped 
> normally it's fine.  But we cannot seem to survive a hard shutdown.  
> We're looking at ways to add a batch script to be called before the 
> server is started, so we can clean up the temp folder.  One idea that 
> seems promising is to replace the wrapper.java.command with a batch 
> file that first cleans the temp folder and then calls java.  Do you 
> have any other suggestions for hooking into application startup?
>
> The batch file would be similar to this:
>
> IF EXIST "%JBOSS_HOME%\server\default
> \tmp" (
>   rmdir /S /Q "%JBOSS_HOME%\server\default\tmp"
> )
>
> "%JAVA_HOME%\bin\java" %*
>
> Thanks again for your help Leif
> Nathan
>
>
> On 10/29/07, *Leif Mortenson* <le...@ta... 
> <mailto:le...@ta...>> wrote:
>
>     Nathan,
>     Java does indeed appear frozen if its memory gets swapped to disk
>     or during long GC cycles.    To an outside process, the JVM is indeed
>     frozen while either event happens.
>
>     Looking at the wrapper.conf that you posted, you have your
>     wrapper.ping.timeout set to 10 seconds.  Why did you set it to a
>     value shorter than the default of 30 seconds?  My first suggestion
>     would be to set the timeout to 60 or 120 seconds and see how that
>     works for you.
>
>     An old version of the Wrapper started to shutdown cleanly when
>     a ping timeout was encountered.  But if the JVM is really frozen,
>     the shutdown would of course still fail.   If the JVM was not really
>     frozen then the JVM would still either be shutdown or killed.  If
>     the "frozen" state is an artifact of GC or of memory swapping, then
>     a better solution is to increase the ping timeout.
>
>     I have used the Wrapper under VMWare quite a bit.  But not in
>     cases where the JVM or VM were under heavy loads.  I have seen
>     problems when the host OS is under heavy loads, but that is not
>     surprising as VMWare becomes pretty doggish under those
>     circumstances.
>
>     One thing that greatly improved the VMWare performance in cases
>     where the host OS was under load was to disable VMWare from
>     allowing the VM's memory to be swapped.  Force it to stay in active
>     memory.  That gets rid of the long freezes.
>
>     Cheers,
>     Leif
>
>     Nathan Wray wrote:
>     >
>     > Hi;
>     >
>     > I have an application being deployed onto JBoss on Windows under
>     > VMware.  JBoss is running as a service under wrapper.
>     > Although the application has run elsewhere for long periods of time
>     > with no issues, we're seeing a problem under wrapper.
>     > Once the app has been running for 1-2 hours, we see:
>     >
>     > ERROR  | wrapper  | 2007/10/26 16:25:17 | JVM appears hung:
>     Timed out
>     > waiting for signal from JVM.
>     > ERROR  | wrapper  | 2007/10/26 16:25:18 | JVM did not exit on
>     request,
>     > terminated
>     >
>     > Our app is deployed as an EAR file, and if the JBoss server process
>     > doesn't shut down cleanly it won't come back up without deleting
>     the
>     > tmp, work and data folders.  So once wrapper has gunned the server
>     > it's dead until someone manually intervenes.
>     >
>     > Using the default ticks timer we see messages like this
>     periodically:
>     >
>     >  The timer fell behind the system clock by 14600ms.
>     >
>     > Our server is a VMware instance, so it is possible that the VM could
>     > be getting swapped out or starved.  This would probably appear
>     to an
>     > application as though the clock and timers were jumping out of
>     sync in
>     > ways that normally are impossible.
>     > Is anyone successfully using wrapper under VMware?
>     >
>     > Also we're running with verbose garbage collection, which I
>     understand
>     > can hang the server momentarily.  Are there known issues with
>     verbose
>     > garbage collection under wrapper?
>     >
>     > Looking through the code it doesn't appear that wrapper tries to
>     shut
>     > down the server once it decides there is a hang; it looks like
>     it goes
>     > right into WRAPPER_JSTATE_KILLING rather than
>     WRAPPER_JSTATE_STOPPING;
>     > so that the shutdownTimeout parameter does not come into
>     play.  Would
>     > it be possible to attempt a clean shutdown before killing the
>     server?
>     >
>     >
>     > Selected settings below.  Note right now we're trying to address the
>     > hard kill that leaves the server unusable, we will eventually
>     bump up
>     > the ping timeout to try and avoid the kill in the first place.
>     >
>     > # Number of seconds between Wrapper ping requests to the JVM.
>     > wrapper.ping.interval=5
>     >
>     > #Number of seconds to allow between the wrapper pinging the JVM and
>     > the response.
>     > wrapper.ping.timeout=10
>     >
>     > #Number of seconds between JVM reports stopped and the JVM actually
>     > terminates.
>     > wrapper.jvm_exit.timeout=60
>     > wrapper.shutdown.timeout=60
>     > wrapper.restart.delay=60
>     >
>     > wrapper.timer_fast_threshold=0
>     > wrapper.timer_slow_threshold=10
>     >
>     > # Force reload of wrapper config on JVM restart
>     > wrapper.restart.reload_configuration=TRUE
>     >
>     > Thanks for any insight you can bring.
>     >
>     > Nathan
>
>
>     -------------------------------------------------------------------------
>     This SF.net email is sponsored by: Splunk Inc.
>     Still grepping through log files to find problems?  Stop.
>     Now Search log events and configuration files using AJAX and a
>     browser.
>     Download your FREE copy of Splunk now >> http://get.splunk.com/
>     _______________________________________________
>     Wrapper-user mailing list
>     Wra...@li...
>     <mailto:Wra...@li...>
>     https://lists.sourceforge.net/lists/listinfo/wrapper-user
>     <https://lists.sourceforge.net/lists/listinfo/wrapper-user>
>
>
>
>
> -- 
> Nathan Wray
> nw...@de... <mailto:nw...@de...>
> -- 
> ------------------------------------------------------------------------
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems?  Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >> http://get.splunk.com/
> ------------------------------------------------------------------------
>
> _______________________________________________
> Wrapper-user mailing list
> Wra...@li...
> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>