|
From: Erik D. <eri...@fj...> - 2009-03-12 15:56:09
|
Hi! Some time after starting the application (20min perhaps) the wrapper seems to invent a problem and kills the jvm. See debug log: INFO | jvm 1 | 2009/03/12 16:28:24 | Send a packet PING : ok DEBUG | wrapperp | 2009/03/12 16:28:24 | read a packet PING : ok DEBUG | wrapper | 2009/03/12 16:28:24 | Got ping response from JVM ERROR | wrapper | 2009/03/12 16:39:26 | JVM exited unexpectedly. DEBUG | wrapper | 2009/03/12 16:39:26 | Signal trapped. Details: DEBUG | wrapper | 2009/03/12 16:39:26 | signal number=17 (SIGCHLD), source="unknown" DEBUG | wrapper | 2009/03/12 16:39:26 | Received SIGCHLD, checking JVM process status. DEBUG | wrapper | 2009/03/12 16:39:26 | JVM process exited with a code of 1, setting the wrapper exit code to 1. INFO | jvm 1 | 2009/03/12 16:39:26 | Read Timed out. (Last Ping was 292200 milliseconds ago) INFO | jvm 1 | 2009/03/12 16:39:26 | Read Timed out. (Last Ping was 585600 milliseconds ago) INFO | jvm 1 | 2009/03/12 16:39:26 | Wrapper Manager: JVM did not exit. Give up. INFO | jvm 1 | 2009/03/12 16:39:26 | Wrapper Manager: ShutdownHook started INFO | jvm 1 | 2009/03/12 16:39:26 | WrapperManager.stop(0) called by thread: Wrapper-Shutdown-Hook INFO | jvm 1 | 2009/03/12 16:39:26 | Send a packet STOP : 0 INFO | jvm 1 | 2009/03/12 16:39:26 | Thread, Wrapper-Shutdown-Hook, handling the shutdown process. INFO | jvm 1 | 2009/03/12 16:39:26 | calling listener.stop() INFO | jvm 1 | 2009/03/12 16:39:26 | WrapperSimpleApp: stop(0) INFO | jvm 1 | 2009/03/12 16:39:26 | returned from listener.stop() -> 0 INFO | jvm 1 | 2009/03/12 16:39:26 | shutdownJVM(0) Thread:Wrapper-Shutdown-Hook INFO | jvm 1 | 2009/03/12 16:39:26 | Send a packet STOPPED : 0 INFO | jvm 1 | 2009/03/12 16:39:26 | Closing socket. INFO | jvm 1 | 2009/03/12 16:39:26 | Wrapper Manager: ShutdownHook complete DEBUG | wrapperp | 2009/03/12 16:39:26 | server listening on port 32000. INFO | wrapper | 2009/03/12 16:39:26 | Wrapper Process has not received any CPU time for 144 seconds. Extending timeouts. DEBUG | wrapper | 2009/03/12 16:39:26 | Waiting 5 seconds before launching another JVM. Any idea what causes this? If I call status I get the response that the application is running, but my application is surely stopped. However, a wrapper process seems to be alive. I have tried extending the timeouts, but this does not seem to effect the problem. wrapper.ping.timeout=300 wrapper.startup.timeout=300 System: Linux 2.6.18-92.el5 #1 SMP i686 i686 i386 GNU/Linux Wrapper: 3.2.3 with WrapperSimpleApp Integration java version "1.6.0_10" Java(TM) SE Runtime Environment (build 1.6.0_10-b33) Java HotSpot(TM) Server VM (build 11.0-b15, mixed mode) -- Best regards, Erik Drolshammer |
|
From: Leif M. <le...@ta...> - 2009-03-12 17:53:21
|
Erik, The Wrapper is receiving a SIGCHLD signal from the OS and interpreting it as the Java process having stopped. From the log however, it appears that JVM still exists as it continues on to shut it down. Can you tell me a little about your application? Does your Java application run any child processes of its own? I am wondering if such a grandchild process exiting might be causing this. I am going to need to go back and run some more tests if that is the case. As a note, when you query the status, it is reporting as running because the Wrapper process itself is still running. At the end of the log, it appears the Wrapper is just about to launch a second JVM invocation to recover from this problem. Cheers, Leif On Fri, Mar 13, 2009 at 12:54 AM, Erik Drolshammer <eri...@fj...> wrote: > Hi! > Some time after starting the application (20min perhaps) the wrapper > seems to invent a problem and kills the jvm. See debug log: > > INFO | jvm 1 | 2009/03/12 16:28:24 | Send a packet PING : ok > DEBUG | wrapperp | 2009/03/12 16:28:24 | read a packet PING : ok > DEBUG | wrapper | 2009/03/12 16:28:24 | Got ping response from JVM > ERROR | wrapper | 2009/03/12 16:39:26 | JVM exited unexpectedly. > DEBUG | wrapper | 2009/03/12 16:39:26 | Signal trapped. Details: > DEBUG | wrapper | 2009/03/12 16:39:26 | signal number=17 (SIGCHLD), > source="unknown" > DEBUG | wrapper | 2009/03/12 16:39:26 | Received SIGCHLD, checking JVM > process status. > DEBUG | wrapper | 2009/03/12 16:39:26 | JVM process exited with a code > of 1, setting the wrapper exit code to 1. > INFO | jvm 1 | 2009/03/12 16:39:26 | Read Timed out. (Last Ping was > 292200 milliseconds ago) > INFO | jvm 1 | 2009/03/12 16:39:26 | Read Timed out. (Last Ping was > 585600 milliseconds ago) > INFO | jvm 1 | 2009/03/12 16:39:26 | Wrapper Manager: JVM did not > exit. Give up. > INFO | jvm 1 | 2009/03/12 16:39:26 | Wrapper Manager: ShutdownHook > started > INFO | jvm 1 | 2009/03/12 16:39:26 | WrapperManager.stop(0) called > by thread: Wrapper-Shutdown-Hook > INFO | jvm 1 | 2009/03/12 16:39:26 | Send a packet STOP : 0 > INFO | jvm 1 | 2009/03/12 16:39:26 | Thread, Wrapper-Shutdown-Hook, > handling the shutdown process. > INFO | jvm 1 | 2009/03/12 16:39:26 | calling listener.stop() > INFO | jvm 1 | 2009/03/12 16:39:26 | WrapperSimpleApp: stop(0) > INFO | jvm 1 | 2009/03/12 16:39:26 | returned from listener.stop() -> 0 > INFO | jvm 1 | 2009/03/12 16:39:26 | shutdownJVM(0) > Thread:Wrapper-Shutdown-Hook > INFO | jvm 1 | 2009/03/12 16:39:26 | Send a packet STOPPED : 0 > INFO | jvm 1 | 2009/03/12 16:39:26 | Closing socket. > INFO | jvm 1 | 2009/03/12 16:39:26 | Wrapper Manager: ShutdownHook > complete > DEBUG | wrapperp | 2009/03/12 16:39:26 | server listening on port 32000. > INFO | wrapper | 2009/03/12 16:39:26 | Wrapper Process has not > received any CPU time for 144 seconds. Extending timeouts. > DEBUG | wrapper | 2009/03/12 16:39:26 | Waiting 5 seconds before > launching another JVM. > > Any idea what causes this? > > If I call status I get the response that the application is running, but > my application is surely stopped. However, a wrapper process seems to be > alive. > > > I have tried extending the timeouts, but this does not seem to effect > the problem. > wrapper.ping.timeout=300 > wrapper.startup.timeout=300 > > > > System: > Linux 2.6.18-92.el5 #1 SMP i686 i686 i386 GNU/Linux > Wrapper: 3.2.3 with WrapperSimpleApp Integration > java version "1.6.0_10" > Java(TM) SE Runtime Environment (build 1.6.0_10-b33) > Java HotSpot(TM) Server VM (build 11.0-b15, mixed mode) > > -- > Best regards, > > Erik Drolshammer |
|
From: Erik D. <eri...@fj...> - 2009-03-17 14:22:20
|
Leif Mortenson wrote: > The Wrapper is receiving a SIGCHLD signal from the OS and interpreting > it as the Java process having stopped. From the log however, it > appears that JVM still exists as it continues on to shut it down. Yes, we have implemented a shutdown hook in the application, so the application shuts down cleanly when it receives the shutdown notification. > Can you tell me a little about your application? Does your Java > application run any child processes of its own? I am wondering if > such a grandchild process exiting might be causing this. I am going > to need to go back and run some more tests if that is the case. No child processes. > As a note, when you query the status, it is reporting as running > because the Wrapper process itself is still running. At the end of > the log, it appears the Wrapper is just about to launch a second JVM > invocation to recover from this problem. It doesn't appear to try to relaunch the app, as far as I can see. As a note, we have the same application deployed on other nodes where this problem does not occur. Which types of environment differences might be related? (The Java version, JSW version and distro is as far as I can tell identical) -- Best regards, Erik Drolshammer |
|
From: Erik D. <eri...@fj...> - 2009-03-18 11:58:02
|
Erik Drolshammer wrote: > Leif Mortenson wrote: >> The Wrapper is receiving a SIGCHLD signal from the OS and interpreting >> it as the Java process having stopped. From the log however, it >> appears that JVM still exists as it continues on to shut it down. So something _outside_ the wrapper and my application is sending a SIGCHLD signal which the wrapper picks up? Any suggestions on how I can debug this further? We are not that comfortable with behavior we cannot explain. It happens rather quickly when put under (light) load, so it is easy to test if you have anything I can try. -- Best regards, Erik Drolshammer |
|
From: Erik D. <eri...@fj...> - 2009-03-27 08:25:35
|
fyi; We did not find the cause of the problem, so we reinstalled the environment as a test. The problem have disappeared. :/ -- Best regards, Erik Drolshammer |