Re: [Wrapper-user] Occasional wrapper restart under light load

[Leif M. <le...@ta...>]

Bill,
    Could you zip up your full unmodified wrapper.log file and send it 
to me directly?
Most likely too big for the list.  I want to compare the output when 
things are going
smoothly to that just before it fails.

    For some reason, the JVM stops replying to ping requests from the 
Wrapper.
After the ping timeout expires, it gives up and restarts the JVM.
    A normal ping cycle looks like the following:
DEBUG  | wrapperp | 2003/10/14 10:43:19 | send a packet 103 : ping
INFO   | jvm 2    | 2003/10/14 10:43:19 | Received a packet 103 : ping
INFO   | jvm 2    | 2003/10/14 10:43:19 | Send a packet 103 : ok
DEBUG  | wrapperp | 2003/10/14 10:43:19 | read a packet 103 : ok
DEBUG  | wrapper  | 2003/10/14 10:43:19 | Got ping response from JVM

    The problem is that before the JVM is restarted, there are no 
messages from
the JVM about having received any packets.
    You are using close to 500MB of memory.  I have seen the JVM take a very
long time to do a single garbage collection sweep.   When this is 
happening, the
%CPU in the task manager does not always show the system as being all that
busy.   If this is the problem you might want to try using incremental 
garbage
collection by adding the -Xincgc.   I was not sure what the
-XX:+UseConcMarkSweepGC option does?
Also try extending your wrapper.ping.timeout to around 300, 5 minutes.  
If the
problem is GC related, that will hopefully be long enough to make the 
problem
go away.   If the problem is GC related, then your application would be
unresponsive to its clients and not just the Wrapper during this time 
however,
have you seen such problems?

    I can't think of anything off hand that I have fixed since version 
3.0.2 that would
affect this, but there have been lots of improvements to the wrapper. 
You may want
to consider upgrading to version 3.0.5

Cheers,
Leif

Bill Littman wrote:

>Hi-
>
>I have an application running on Windows 2000. It has dual Xeons and 2
>Gigs of memory. I am using Sun Java JRE 1.4.1_02 and Wrapper version
>3.0.2.
>
>My application is a CORBA server serving data from a local DB2 database.
>The ORB we use is OpenORB 1.3.0.
>
>Very occasionally the wrapper restarts the application while the
>application is under very little load. There is no reason I can think of
>why it would do this. I am pretty sure that the application is not
>running out of resources. Memory usage appears to be fine and the number
>of allocated threads appears bounded. The amount of time from
>application start to when this problem occurs is not constant and can
>vary wildly.
>
>I have attached my config files. A few notes on the config files:
>
>1. When this started, I added the lines: wrapper.ping.timeout=180 and
>wrapper.cpu.timeout=30. This restart occurs so infrequently that it is
>impossible to tell if this helps. As the last restart shows, it does not
>solve the problem.
>
>2. Just recently, I changed the wrapper.logfile.loglevel to DEBUG and
>added the line: wrapper.request_thread_dump_on_failed_jvm_exit=TRUE.
>
>3. One of the JVM flags, -Xloggc:C:\Tomo\Logs\GC.log, sends garbage
>collection traces out to a log. There is a JVM bug where some of the
>messages are sent to the console instead of the log file. Those messages
>end up in the wrapper log produced by the application.
>
>So, from the attached wrapper log, here is the last two pings followed
>by the error handling traces, with the garbage collection traces
>removed.
>
>DEBUG  | wrapperp | 2003/10/14 10:42:51 | send a packet 103 : ping
>DEBUG  | wrapperp | 2003/10/14 10:42:57 | send a packet 103 : ping
>ERROR  | wrapper  | 2003/10/14 10:43:03 | JVM appears hung: Timed out
>waiting for signal from JVM.
>STATUS | wrapper  | 2003/10/14 10:43:03 | Dumping JVM state.
>DEBUG  | wrapper  | 2003/10/14 10:43:03 | Sending BREAK event to process
>group 2492.
>ERROR  | wrapper  | 2003/10/14 10:43:03 | Unable to send BREAK event to
>JVM process.  Err(6 : The handle is invalid. (0x6))
>ERROR  | wrapper  | 2003/10/14 10:43:04 | Java Virtual Machine did not
>exit on request, terminated
>STATUS | wrapper  | 2003/10/14 10:43:10 | Launching a JVM...
>
>I am not sure what this means. Can someone help?
>
>Thank you.
>
>-Bill Littman
> Lead Software Engineer
> TomoTherapy, Inc.
> 1240 Deming Way
> Madison, WI  53717
> Direct Phone: 608 824-2815
> Phone: 608 824-2800
> Fax: 608 824-2996
> Web address: http://www.tomotherapy.com
> Email: bli...@to...
>  
>