Wrapper in Zombie state

[Muhammad Jamil Chaudhary]

Hi,

We did OS upgrade from 6.5 to 6.9 of Linux. After upgrade few of our applications start behaving strange. Wrapper Version is 3.5.26, JDK version 1.7.0_71.

It looks JVM stuck on shut down until manualy stop the service.
In the stack trace of wrapper i can see which is not giving any helpfull detail.

Log detail of wrapper:

ERROR | wrapper | 2019/01/21 10:32:45 | JVM did not exit on request, termination requested.
DEBUG | wrapper | 2019/01/21 10:32:45 | Signal trapped. Details:
DEBUG | wrapper | 2019/01/21 10:32:45 | signal number=17 (SIGCHLD), source="unknown"
DEBUG | wrapper | 2019/01/21 10:32:45 | Received SIGCHLD, checking JVM process status.
STATUS | wrapper | 2019/01/21 10:32:45 | JVM received a signal SIGKILL (9).
STATUS | wrapper | 2019/01/21 10:32:45 | JVM process is gone.
DEBUG | wrapper | 2019/01/21 10:32:45 | JVM process exited with a code of 1, setting the wrapper exit code to 1.
STATUS | wrapper | 2019/01/21 10:32:45 | JVM exited after being requested to terminate.
DEBUG | wrapper | 2019/01/21 10:32:45 | Preparing to restart with mode 2.
DEBUG | wrapper | 2019/01/21 10:32:45 | Waiting 60 seconds before launching another JVM.
STATUS | wrapper | 2019/01/21 10:33:45 | Reloading Wrapper configuration...
DEBUG | wrapper | 2019/01/21 10:33:45 | Working directory set to: /apps/HOME/engines/wrapper/3.5.26/bin
DEBUG | wrapper | 2019/01/21 10:33:45 | Working directory set to: /apps/HOME/engines/CDRExport
DEBUG | wrapper | 2019/01/21 10:33:45 | Working directory set to: /apps/HOME/engines/CDRExport
DEBUG | wrapper | 2019/01/21 10:33:45 | Resolved the real path of wrapper.java.command as an absolute reference: /java/jdk1.7.0_71/bin/java
DEBUG | wrapper | 2019/01/21 10:33:45 | Magic number for file /java/jdk1.7.0_71/bin/java: 0x7f454c46
DEBUG | wrapper | 2019/01/21 10:33:45 | Startup Timeouts: wrapper.startup.timeout=30, wrapper.startup.delay.console=0, wrapper.startup.delay.service=0, wrapper.restart.delay=60
DEBUG | wrapper | 2019/01/21 10:33:45 | Ping settings: wrapper.ping.interval=5, wrapper.ping.interval.logged=1, wrapper.ping.timeout=30, wrapper.ping.alert.threshold=1
DEBUG | wrapper | 2019/01/21 10:33:45 | Shutdown Timeouts: wrapper.shutdown.timeout=30, wrapper.jvm_exit.timeout=15, wrapper.jvm_cleanup.timeout=10, wrapper.jvm_terminate.timeout=10
DEBUG | wrapperp | 2019/01/21 10:33:45 | Send a packet LOGFILE : wrapper.log
DEBUG | wrapperp | 2019/01/21 10:33:45 | Socket send failed. Bad file descriptor
DEBUG | wrapper | 2019/01/21 10:33:45 | active log file changed: wrapper.log
STATUS | wrapper | 2019/01/21 10:33:45 | Launching a JVM...
DEBUG | wrapper | 2019/01/21 10:33:45 | Java Command Line:
DEBUG | wrapper | 2019/01/21 10:33:45 | Command[0] :/java/jdk1.7.0_71/bin/java
DEBUG | wrapper | 2019/01/21 10:33:45 | Command[1] : -Xms10m
DEBUG | wrapper | 2019/01/21 10:33:45 | Command[2] : -Xmx2048m
DEBUG | wrapper | 2019/01/21 10:33:45 | Command[3] : -Djava.library.path=/apps/HOME/engines/wrapper/3.5.26/lib
DEBUG | wrapper | 2019/01/21 10:33:45 | Command[4] : -classpath
DEBUG | wrapper | 2019/01/21 10:33:45 | Command[5] : /apps/HOME/engines/lib-bootstrap:/apps/HOME/engines/lib-bootstrap/HOME/lib/advanced-queuing.jar:/apps/HOME/engines/lib-bootstrap/HOME/bootstrap_persistence.jar:/apps/HOME/engines/lib-bootstrap/HOME/lib/management_console-commands.jar:/apps/HOME/engines/lib-bootstrap/HOME/lib/management_console-core.jar:/apps/HOME/engines/lib-bootstrap/HOME/lib/management_console-engine_framework.jar:/apps/HOME/engines/lib-bootstrap:/apps/HOME/engines/lib-bootstrap/HOME/lib/3rdparty/wrapper.jar
DEBUG | wrapper | 2019/01/21 10:33:45 | Command[6] : -Dwrapper.key=NjQn3cUnOjtNFWQoJgsAlIP_jPKsIlgY
DEBUG | wrapper | 2019/01/21 10:33:45 | Command[7] : -Dwrapper.port=32037
DEBUG | wrapper | 2019/01/21 10:33:45 | Command[8] : -Dwrapper.debug=TRUE
DEBUG | wrapper | 2019/01/21 10:33:45 | Command[9] : -Dwrapper.disable_console_input=TRUE
DEBUG | wrapper | 2019/01/21 10:33:45 | Command[10] : -Dwrapper.pid=4967
DEBUG | wrapper | 2019/01/21 10:33:45 | Command[11] : -Dwrapper.version=3.5.26-st
DEBUG | wrapper | 2019/01/21 10:33:45 | Command[12] : -Dwrapper.native_library=wrapper
DEBUG | wrapper | 2019/01/21 10:33:45 | Command[13] : -Dwrapper.arch=x86
DEBUG | wrapper | 2019/01/21 10:33:45 | Command[14] : -Dwrapper.service=TRUE
DEBUG | wrapper | 2019/01/21 10:33:45 | Command[15] : -Dwrapper.cpu.timeout=10
DEBUG | wrapper | 2019/01/21 10:33:45 | Command[16] : -Dwrapper.jvmid=794
DEBUG | wrapper | 2019/01/21 10:33:45 | Command[17] : -Dwrapper.lang.domain=wrapper
DEBUG | wrapper | 2019/01/21 10:33:45 | Command[18] : -Dwrapper.lang.folder=../lang
DEBUG | wrapper | 2019/01/21 10:33:45 | Command[19] : com.billing.engine.GenericEngineStarter
DEBUG | wrapper | 2019/01/21 10:33:45 | Command[20] : -R
DEBUG | wrapper | 2019/01/21 10:33:45 | Command[21] : CDRExport
DEBUG | wrapper | 2019/01/21 10:33:45 | Command[22] : -P
INFO | jvm 794 | 2019/01/21 10:33:45 | GenericEngineStarter.main()
INFO | jvm 794 | 2019/01/21 10:33:45 | bootstrap properties file used: file:/HOME/engines/lib-bootstrap/bootstrap.properties
ERROR | wrapper | 2019/01/21 10:34:14 | Startup failed: Timed out waiting for a signal from the JVM.
ADVICE | wrapper | 2019/01/21 10:34:14 |
ADVICE | wrapper | 2019/01/21 10:34:14 | ------------------------------------------------------------------------
ADVICE | wrapper | 2019/01/21 10:34:14 | Advice:
ADVICE | wrapper | 2019/01/21 10:34:14 | The Wrapper consists of a native component as well as a set of classes
ADVICE | wrapper | 2019/01/21 10:34:14 | which run within the JVM that it launches. The Java component of the
ADVICE | wrapper | 2019/01/21 10:34:14 | Wrapper must be initialized promptly after the JVM is launched or the
ADVICE | wrapper | 2019/01/21 10:34:14 | Wrapper will timeout, as just happened. Most likely the main class
ADVICE | wrapper | 2019/01/21 10:34:14 | specified in the Wrapper configuration file is not correctly initializing
ADVICE | wrapper | 2019/01/21 10:34:14 | the Wrapper classes:
ADVICE | wrapper | 2019/01/21 10:34:14 | com.billing.engine.GenericEngineStarter
ADVICE | wrapper | 2019/01/21 10:34:14 | While it is possible to do so manually, the Wrapper ships with helper
ADVICE | wrapper | 2019/01/21 10:34:14 | classes to make this initialization processes automatic.
ADVICE | wrapper | 2019/01/21 10:34:14 | Please review the integration section of the Wrapper's documentation
ADVICE | wrapper | 2019/01/21 10:34:14 | for the various methods which can be employed to launch an application
ADVICE | wrapper | 2019/01/21 10:34:14 | within the Wrapper:
ADVICE | wrapper | 2019/01/21 10:34:14 | http://wrapper.tanukisoftware.com/doc/english/integrate.html
ADVICE | wrapper | 2019/01/21 10:34:14 | ------------------------------------------------------------------------
ADVICE | wrapper | 2019/01/21 10:34:14 |
ERROR | wrapper | 2019/01/21 10:34:14 | JVM did not exit on request, termination requested.
DEBUG | wrapper | 2019/01/21 10:34:14 | Signal trapped. Details:
DEBUG | wrapper | 2019/01/21 10:34:14 | signal number=17 (SIGCHLD), source="unknown"
DEBUG | wrapper | 2019/01/21 10:34:14 | Received SIGCHLD, checking JVM process status.
STATUS | wrapper | 2019/01/21 10:34:14 | JVM received a signal SIGKILL (9).
STATUS | wrapper | 2019/01/21 10:34:14 | JVM process is gone.
DEBUG | wrapper | 2019/01/21 10:34:14 | JVM process exited with a code of 1, setting the wrapper exit code to 1.
STATUS | wrapper | 2019/01/21 10:34:14 | JVM exited after being requested to terminate.
DEBUG | wrapper | 2019/01/21 10:34:14 | Preparing to restart with mode 2.
DEBUG | wrapper | 2019/01/21 10:34:14 | Waiting 60 seconds before launching another JVM.

[Leif Mortenson]

Are you attempting to use Integration method 3? This would mean that your GenericEngineStarter class is directly or indirectly implementing the WrapperListener interface.
In this case you need to be starting up the WrapperManager and then calling start with the WrapperListener instance. It is then important that your WrapperListener.start method always return in a timely manner. Please see the following page for details:
https://wrapper.tanukisoftware.com/doc/english/integrate-listener.html
The error you see will happen if the WrapperManager is never set up correctly or if the WrapperListener.start method is never returned.

In general we recommend using one of the more simple integration methods (1, 2, or 4) unless absolutely necessary.
https://wrapper.tanukisoftware.com/doc/english/integrate.html

I have a question for you though.
It looks like all of your Java side log output is being sent to a different log tool, so nothing is being sent to the console.
How and why are you doing this? The wrapper has a number of useful features which rely on this output so I want to understand what you are doing to help with our design of the product.

Thanks,
Leif

[Muhammad Jamil Chaudhary]

Hi Lief,

Thank you for your kind response.
Yes we are using 3 and made a class GenericEngineStarter to use in our applications.

I am looking the class how our developer implemented the GenericEngineStarter and WrapperManager class.

Answer to your question is YES we are using few tools to get the detail from the log of the applications to triggering alerts and doing some statistics.

Is there any easy way to control shut down and start of application in the hanging state

Should I consider to use lated version of wrapper 3.5.36?

here are few we are already configured

wrapper.on_exit.default=RESTART

wrapper.on_exit.0=RESTART

wrapper.on_exit.99=SHUTDOWN

wrapper.ntservice.process_priority=LOW

wrapper.ping.interval=5

wrapper.ping.timeout=30

wrapper.ping.timeout.action=DUMP,RESTART

wrapper.ping.alert.threshold=1

[Leif Mortenson]

Muhammad
Please be sure to read over the Method 3 documentation.
This is custom code on your end so it is not possible to help much without seeing the source.
But there is nothing in the log about the WrapperManager being initialized and the notice that you are seeing happens when the WrapperManager and your WrapperListener.start method complete. So most likely the Java code is not written correctly..
Nothing about the Wrapper or Linux upgrade should have affected this. Maybe your code has different logic based on the environment?
Please add some debug output to your code and make sure that

---

WrapperManager.start( new Main(), args );

---

is being called, and that your WrapperListener.start is being called and returns a value.

Cheers,
Leif