|
From: Wendell M. <mac...@gm...> - 2017-01-25 19:50:43
|
sounds like a system management process is determining that your JVM instance is consuming TOO much system resource and it is being terminated with the kill -9. try keep a running snapshot of system resources every 5 minutes or so to a log and see if there is a spike occuring at some point prior to the termination of the VM. Wendell On Wed, Jan 25, 2017 at 11:06 AM, Paul Manning <Pau...@us...> wrote: > All, > > > > We are currently running an application using Tanuki 3.2.3 with > ServiceMix. > > > > The application seems to randomly fail with the following error in the > wrapper log: > > > > ERROR | wrapper | 2017/01/24 13:00:06 | JVM exited unexpectedly. > > DEBUG | wrapper | 2017/01/24 13:00:06 | Signal trapped. Details: > > DEBUG | wrapper | 2017/01/24 13:00:06 | signal number=17 (SIGCHLD), > source="unknown" > > DEBUG | wrapper | 2017/01/24 13:00:06 | Received SIGCHLD, checking JVM > process status. > > STATUS | wrapper | 2017/01/24 13:00:06 | JVM exited in response to signal > SIGKILL (9). > > DEBUG | wrapper | 2017/01/24 13:00:06 | JVM process exited with a code > of 1, setting the wrapper exit code to 1. > > > > We currently have Tanuki setup to auto restart, so the application comes > back online immediately. However, it will eventually crash again. Sometimes > this is within hours or sometimes within days. > > > > *Notes*: > > The application is running with Java 1.7.0_45 on Redhat Linux 64 bit. > > There are no error messages or exceptions in our application logs > (ServiceMix). > > There are no JVM crash reports (hs_err_pid files). > > There are no core dumps. > > This is not being killed by the Linux OOM Killer or another user. The > linux system logs are clean. > > No other exceptions in the wrapper log. > > Verbose GC Logging is enabled and the system memory is healthy. Memory is > about 50% utilization when the crash recently occurred. > > Thread dumps are being taken periodically and they look clean (no > deadlock). > > This is a commercial application that is running at many customer sites > for many years and we have never seen a problem like this before. > > I cannot reproduce this problem in house and I have limited access to the > production system (thru a 3rd party). > > > > Is there anything that we can do to get more intel on why this process is > crashing? > > > > Thanks, > > > > Paul Manning > > > > > > > > ------------------------------------------------------------ > ------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, SlashDot.org! http://sdm.link/slashdot > _______________________________________________ > Wrapper-user mailing list > Wra...@li... > https://lists.sourceforge.net/lists/listinfo/wrapper-user > > -- D. Wendell MacKenzie Software Developer Email: mac...@gm... Phone: 902-626-8708 (cell) |