#4492 [HHQ-4359] When server runs out of memory stop command doesn

[Hyperic Team]

http://jira.hyperic.com/browse/HHQ-4359
Symptoms of the issue:

HQ server ran out of memory (because of postgres DB connection pool was full and most likely data inserter queue caused in memory cache bloating)


I increased JVM heap and max connection parameters and ran HQ server restart


Both stop and start actions ran successfully but stop failed internally. It failed to stop Catalina bootstrap process.

From wrapper.log:

STATUS | wrapper | 2010/10/07 01:42:05 | TERM trapped. Shutting down.


ERROR | wrapper | 2010/10/07 01:42:39 | Shutdown failed: Timed out waiting for signal from JVM.


ERROR | wrapper | 2010/10/07 01:42:39 | JVM did not exit on request, terminated


INFO | wrapper | 2010/10/07 01:42:39 | JVM exited on its own while waiting to kill the application.


STATUS | wrapper | 2010/10/07 01:42:40 | JVM exited in response to signal SIGKILL (9).


STATUS | wrapper | 2010/10/07 01:42:40 | <-- Wrapper Stopped

Though wrapper log tells that the server jvm process was killed I still see catalina bootstrap process running (note the process start date/time from process table output). It's 10/6 6:48, the SIGKILL was sent on 10/7 1:42 (from wrapper log above)

[hyperic@vmc-ssrc-rh94 server-4.5.0.BUILD-SNAPSHOT-EE]$ ps -elf|grep apache


0 S hyperic 19202 4095 0 78 0 - 15288 pipe_w 01:55 pts/2 00:00:00 grep apache


0 S hyperic 20263 1 31 78 0 - 381580 184467 Oct06 ? 06:48:23 /home/hyperic/PostgreSQL/112/server-4.5.0.BUILD-SNAPSHOT-EE/jre/bin/java -cp /home/hyperic/PostgreSQL/112/server-4.5.0.BUILD-SNAPSHOT-EE/hq-engine/hq-server/bin/tomcat-juli.jar:/home/hyperic/PostgreSQL/112/server-4.5.0.BUILD-SNAPSHOT-EE/hq-engine/tomcat-6.0.29.A.RELEASE/bin/bootstrap.jar -XX:MaxPermSize=192m -Xmx512m -Xms512m -XX:+HeapDumpOnOutOfMemoryError -Dserver.home=/home/hyperic/PostgreSQL/112/server-4.5.0.BUILD-SNAPSHOT-EE -Dcatalina.config=file:/home/hyperic/PostgreSQL/112/server-4.5.0.BUILD-SNAPSHOT-EE/hq-engine/hq-server/conf/hq-catalina.properties -Dcom.sun.management.jmxremote -Djava.endorsed.dirs/home/hyperic/PostgreSQL/112/server-4.5.0.BUILD-SNAPSHOT-EE/hq-engine/tomcat-6.0.29.A.RELEASE/common/endorsed -Dcatalina.base=/home/hyperic/PostgreSQL/112/server-4.5.0.BUILD-SNAPSHOT-EE/hq-engine/hq-server -Dcatalina.home=/home/hyperic/PostgreSQL/112/server-4.5.0.BUILD-SNAPSHOT-EE/hq-engine/tomcat-6.0.29.A.RELEASE -Djava.io.tmpdir=/home/hyperic/PostgreSQL/112/server-4.5.0.BUILD-SNAPSHOT-EE/hq-engine/hq-server/temp -Djava.util.logging.manager=com.springsource.tcserver.serviceability.logging.TcServerLogManager org.apache.catalina.startup.Bootstrap start