[Wrapper-user] JVM Exited with No details - Need Help

[Paul M. <Pau...@us...>]

All,

We are currently running an application using Tanuki 3.2.3 with ServiceMix.

The application seems to randomly fail with the following error in the wrapper log:

ERROR  | wrapper  | 2017/01/24 13:00:06 | JVM exited unexpectedly.
DEBUG  | wrapper  | 2017/01/24 13:00:06 | Signal trapped.  Details:
DEBUG  | wrapper  | 2017/01/24 13:00:06 |   signal number=17 (SIGCHLD), source="unknown"
DEBUG  | wrapper  | 2017/01/24 13:00:06 | Received SIGCHLD, checking JVM process status.
STATUS | wrapper  | 2017/01/24 13:00:06 | JVM exited in response to signal SIGKILL (9).
DEBUG  | wrapper  | 2017/01/24 13:00:06 | JVM process exited with a code of 1, setting the wrapper exit code to 1.

We currently have Tanuki setup to auto restart, so the application comes back online immediately. However, it will eventually crash again. Sometimes this is within hours or sometimes within days.

Notes:
The application is running with Java 1.7.0_45 on Redhat Linux 64 bit.
There are no error messages or exceptions in our application logs (ServiceMix).
There are no JVM crash reports (hs_err_pid files).
There are no core dumps.
This is not being killed by the Linux OOM Killer or another user. The linux system logs are clean.
No other exceptions in the wrapper log.
Verbose GC Logging is enabled and the system memory is healthy. Memory is about 50% utilization when the crash recently occurred.
Thread dumps are being taken periodically and they look clean (no deadlock).
This is a commercial application that is running at many customer sites for many years and we have never seen a problem like this before.
I cannot reproduce this problem in house and I have limited access to the production system (thru a 3rd party).

Is there anything that we can do to get more intel on why this process is crashing?

Thanks,

Paul Manning