#173 Service is not stopping properly on linux

[Arushi Rai]

YAJSW: yajsw-stable-12.12
OS : Linux/4.19.115-9.ph3/amd64
JVM : Azul Systems, Inc./11.0.7//usr/java/jre-vmware/64

Stopping the service using -p is not working properly.
When I do systemctl status <service>, it says service "failed" instead of "inactive" </service>

~~~
failed (Result: exit-code) since Thu 2020-07-16 01:29:41 UTC; 5s ago
Docs: man:systemd-sysv-generator(8)
Process: 16621 ExecStart=/etc/rc.d/init.d/ser start (code=exited, status=0/SUCCESS)
Main PID: 16652 (code=exited, status=143)
Main process exited, code=exited, status=143/n/a
Failed with result 'exit-code'.

snippet of wrapper configuration: 

**********

Wrapper Posix Daemon Properties

**********

Directory in which to create and execute daemon init scripts. Default: /etc/init.d

wrapper.daemon.dir =/etc/init.d

Directory where to store the wrapper pid file. Default: /var/run

wrapper.daemon.pid.dir =${base}

Directory in which to create K... and S... links.

Default: <wrapper.daemon.dir>/rcX.d</wrapper.daemon.dir>

For Ubuntu set to /etc/rcX.d

The following grooy script should set it correctly for most distros

wrapper.daemon.run_level_dir=${if (new File('\/etc\/rc0.d').exists()) return '\/etc\/rcX.d' else return '\/etc\/init.d\/rcX.d'}

Setting the runlevels and priorities for automatic startup and stop of the daemon.

Similar syntax as the update_rc.d command

Default: equivalent to default of update_rc.d

wrapper.daemon.update_rc = start 002 3 4 5 . stop 002 0 1 2 6

**********

Exit Code Properties

Restart on non zero exit code

**********

wrapper.on_exit.0=SHUTDOWN
wrapper.on_exit.default=RESTART

wrapper.app.account= \gateway

**********

Wrapper timeouts

**********

Number of seconds to allow between the time that the Wrapper starts the application

and the application logon to the wrapper.

Default: 30 seconds

wrapper.startup.timeout = 120

Number of seconds to allow between the time that the Wrapper asks the application to shutdown and the time that the JVM shuts down.

Default: 30 seconds

wrapper.shutdown.timeout = 120

wrapper.debug = true
wrapper.debug.level = 3

Snippet of wrapper log: 

INFO|wrapper|Service X|20-07-16 01:29:34|Shutting down Wrapper
INFO|wrapper|Service X|20-07-16 01:29:34|stop from Thread Thread-3 reason: USER
INFO|wrapper|Service X|20-07-16 01:29:34|stop from Thread Thread-2 reason: WRAPPER SHUTDOWN
INFO|wrapper|Service X|20-07-16 01:29:34|set state RUNNING->STATE_USER_STOP
INFO|wrapper|Service X|20-07-16 01:29:34|stopping process with pid/timeout 16675 135000
INFO|16675/0|Service X|20-07-16 01:29:34|Controller State: LOGGED_ON -> USER_STOP
INFO|16675/0|Service X|20-07-16 01:29:34|controller sending a stop command
INFO|16675/0|Service X|20-07-16 01:29:34|controller sending a stop command
INFO|16675/0|Service X|20-07-16 01:29:34|controller sending a stop command
INFO|16675/0|Service X|20-07-16 01:29:35|controller close session
INFO|16675/0|Service X|20-07-16 01:29:35|Controller State: USER_STOP -> WAITING_CLOSED
INFO|16675/0|Service X|20-07-16 01:29:35|session closed -> waiting
INFO|16675/0|Service X|20-07-16 01:29:35|unexpected connection closed: /127.0.0.1:24160
INFO|16675/0|Service X|20-07-16 01:29:35|wrapper manager received stop command
INFO|wrapper|Service X|20-07-16 01:29:35|waitpid 16675 0
INFO|wrapper|Service X|20-07-16 01:29:35|exit code posix process: 0 application(status/signal): 0/0
INFO|16675/0|Service X|20-07-16 01:29:35|process terminated
INFO|16675/0|Service X|20-07-16 01:29:35|Controller State: WAITING_CLOSED -> PROCESS_KILLED
INFO|wrapper|Service X|20-07-16 01:29:35|giving up after 0 retries
INFO|wrapper|Service X|20-07-16 01:29:35|set state STATE_USER_STOP->IDLE
INFO|wrapper|Service X|20-07-16 01:29:35|shutdown wrapper due to exit code rule
INFO|wrapper|Service X|20-07-16 01:29:35|stop from Thread yajsw.controller-1 reason: WRAPPER SHUTDOWN
INFO|wrapper|Service X|20-07-16 01:29:35|killing 16675
INFO|wrapper|Service X|20-07-16 01:29:36|process exit code: 0
INFO|wrapper|Service X|20-07-16 01:29:36|removed pid file /opt///X/wrapper.X.pid
INFO|wrapper|Service X|20-07-16 01:29:36|set state IDLE->STATE_SHUTDOWN
INFO|wrapper|Service X|20-07-16 01:29:36|set state IDLE->STATE_SHUTDOWN
INFO|16675/0|Service X|20-07-16 01:29:36|gobler terminated ERROR 16675
INFO|16675/0|Service X|20-07-16 01:29:36|gobler terminated OUTPUT 16675
~~~

Any suggestion on how to troublehsoot this further?

[rzo]

hello,

this is not a shutdown bug. according to the log the application has been shut down correctly.

invocation of /etc/init.d/<service> status
returns a 0 if the service is running and a 3 otherwise

systemctl seems to be expecting a different value.
to query the status of the service use the provided command:
  <yajsw>/bin/queryDaemon.sh

-- Ron

[Arushi Rai]

Sure Ron, let me try that.
Also, we do have another service running on this server using YAJSW and we did not observe this behavior. Let me dig in to see the difference between the two.

Thank you for your response. I might reach out to you again if needed :)

[Arushi Rai]

This may be the reason - https://stegard.net/2016/08/gracefully-killing-a-java-process-managed-by-systemd/

[Arushi Rai]

I updated the unit file for the service
[Service]
SuccessExitStatus=143
https://serverfault.com/questions/695849/services-remain-in-failed-state-after-stopped-with-systemctl

Now when I stop the service, it shows as 'inactive' (expected behavior)

Is there any such configuration in yajsw wrapper?

[rzo]

release 12.15 now supports systemd