[Wrapper-user] AIX only: Wrapper receives an immediate request to shutdown from the JVM

[clvrmnky <clv...@gm...>]

Wrapper 3.1.2
AIX 5.2
Java 5

Similar to <http://sourceforge.net/mailarchive/message.php?msg_id=3D9831093=
>,
I am having a problem getting our JVM to launch our JBoss app.  This
is only a problem on AIX for us.

I'm working on ensuring the correct APARs for AIX to run a Java 5 RC 1
VM have been applied, in case this is a VM issue (so far, I think the
system is patched to the level it needs to be).  I _can_ run and
launch the sample app at version 3.1.2 I downloaded from SourceForge,
though when I hack in the sample wrapper executable into our app I get
the same problem.  I've attached the output from wrapper.debug=3Dtrue.

Now, to be perfectly clear, we recompile wrapper from scratch so the
service name is prettier and matches our product name (here changed to
"fnord").  The names of the login, company and product have been
changed to protect the potentially stupid.  Namely, me.

Notes on the .conf file.  These were added to help debug things, and
do not change the behaviour I'm seeing here:
fnord.java.additional.21=3D-verbose:jni
fnord.java.additional.22=3D-Xcheck:jni
fnord.java.additional.23=3D-Dorg.tanukisoft.wrapper.WrapperSimpleApp.maxSta=
rtMainWait=3D30

Can anyone shed some light on this?  I accept that it could be the VM
(IBM has a wonderfully tweaked VM that can be an utter joy to work
with), but all I can see is that wrapperProtocolRead() reads
WRAPPER_MSG_STOP.  Advice on how to find out why would be much
appreciated.

One note is that I cannot simply update Wrapper to some other revision
unless I am really sure it will solve the problem.  We have a pretty
tight control on changes this late in the release cycle, and every
other Unix system we support seems to work fine.  Just FYI.

--
clvrmnky <mailto:clv...@gm...>

[startup.log]
STATUS | 2006/03/30 12:26:37 | --> Fnord World Domination Server 2006
Started as Console
DEBUG  | 2006/03/30 12:26:37 | Using system timer.
DEBUG  | 2006/03/30 12:26:37 | server listening on port 32000.
DEBUG  | 2006/03/30 12:26:37 | Command[0] :
/home/clvrmnky/servers/3817aix/jre/bin/java
DEBUG  | 2006/03/30 12:26:37 | Command[1] : -Xms512m
DEBUG  | 2006/03/30 12:26:37 | Command[2] : -Xmx512m
DEBUG  | 2006/03/30 12:26:37 | Command[3] :
-Xbootclasspath/p:/home/clvrmnky/servers/3817aix/server/fnord/patch/patchfn=
ordboot/classes:/home/clvrmnky/servers/3817aix/server/fnord/patch/patchfnor=
dboot.jar:/home/clvrmnky/servers/3817aix/server/fnord/lib/boot/fnordboot.ja=
r
DEBUG  | 2006/03/30 12:26:37 | Command[4] : -Xss256k
DEBUG  | 2006/03/30 12:26:37 | Command[5] : -Dprogram.name=3Drun.bat
DEBUG  | 2006/03/30 12:26:37 | Command[6] : -Djboss.identity.domain=3DFNORD
DEBUG  | 2006/03/30 12:26:37 | Command[7] : -Dfnord.safemode=3Dfalse
DEBUG  | 2006/03/30 12:26:37 | Command[8] : -Dfnord.safemode.password=3D<se=
cret>
DEBUG  | 2006/03/30 12:26:37 | Command[9] : -Dfnord.unattended.startup=3Dtr=
ue
DEBUG  | 2006/03/30 12:26:37 | Command[10] : -Dfnord.master.key=3Dfake_mast=
er_key
DEBUG  | 2006/03/30 12:26:37 | Command[11] : -Dfnord.dynamic.aop=3Dfalse
DEBUG  | 2006/03/30 12:26:37 | Command[12] : -Dxmbean.metadata.xml.validate=
=3Dtrue
DEBUG  | 2006/03/30 12:26:37 | Command[13] :
-Dfnordis.neurondir=3D/home/clvrmnky/servers/3817aix/server/fnord/bin
DEBUG  | 2006/03/30 12:26:37 | Command[14] :
-Djboss.server.base.dir=3D/home/clvrmnky/servers/3817aix/server
DEBUG  | 2006/03/30 12:26:37 | Command[15] :
-Djboss.server.base.url=3Dfile:///home/clvrmnky/servers/3817aix/server
DEBUG  | 2006/03/30 12:26:37 | Command[16] :
-Djava.endorsed.dirs=3D/home/clvrmnky/servers/3817aix/server/lib/endorsed
DEBUG  | 2006/03/30 12:26:37 | Command[17] : -Djava.awt.headless=3Dtrue
DEBUG  | 2006/03/30 12:26:37 | Command[18] :
-Dfnord.installDir=3D/home/clvrmnky/servers/3817aix
DEBUG  | 2006/03/30 12:26:37 | Command[19] : -Dfnord.install=3Dserver
DEBUG  | 2006/03/30 12:26:37 | Command[20] : -verbose:jni
DEBUG  | 2006/03/30 12:26:37 | Command[21] : -Xcheck:jni
DEBUG  | 2006/03/30 12:26:37 | Command[22] :
-Dorg.tanukisoft.wrapper.WrapperSimpleApp.maxStartMainWait=3D300
DEBUG  | 2006/03/30 12:26:37 | Command[23] :
-Djava.library.path=3D/home/clvrmnky/servers/3817aix/server/lib:/home/clvrm=
nky/servers/3817aix/server/fnord/bin
DEBUG  | 2006/03/30 12:26:37 | Command[24] : -classpath
DEBUG  | 2006/03/30 12:26:37 | Command[25] :
/home/clvrmnky/servers/3817aix/server/lib/fnordservice.jar:/home/clvrmnky/s=
ervers/3817aix/server/bin/run.jar
DEBUG  | 2006/03/30 12:26:37 | Command[26] : -Dwrapper.key=3DFyIsmbXc06V1kO=
_V
DEBUG  | 2006/03/30 12:26:37 | Command[27] : -Dwrapper.port=3D32000
DEBUG  | 2006/03/30 12:26:37 | Command[28] : -Dwrapper.debug=3DTRUE
DEBUG  | 2006/03/30 12:26:37 | Command[29] : -Dwrapper.use_system_time=3DTR=
UE
DEBUG  | 2006/03/30 12:26:37 | Command[30] : -Dwrapper.version=3D3.1.2
DEBUG  | 2006/03/30 12:26:37 | Command[31] :
-Dwrapper.native_library=3Dfnordservice
DEBUG  | 2006/03/30 12:26:37 | Command[32] : -Dwrapper.cpu.timeout=3D10
DEBUG  | 2006/03/30 12:26:37 | Command[33] : -Dwrapper.jvmid=3D1
DEBUG  | 2006/03/30 12:26:37 | Command[34] :
org.tanukisoftware.wrapper.WrapperSimpleApp
DEBUG  | 2006/03/30 12:26:37 | Command[35] : org.jboss.Main
DEBUG  | 2006/03/30 12:26:37 | Command[36] : -c
DEBUG  | 2006/03/30 12:26:37 | Command[37] : fnord
STATUS | 2006/03/30 12:26:37 | Launching a JVM...
INFO   | 2006/03/30 12:26:38 | JVMJNCK001I JNI check utility
installed. Use -Xcheck:jni:help for usage
INFO   | 2006/03/30 12:26:44 | WrapperManager class initialized by
thread: main  Using classloader:
sun.misc.Launcher$AppClassLoader@2ff02ff0
INFO   | 2006/03/30 12:26:44 | Wrapper Manager: JVM #1
INFO   | 2006/03/30 12:26:44 | Wrapper Manager: Registering shutdown hook
INFO   | 2006/03/30 12:26:44 | Wrapper Manager: Using wrapper
INFO   | 2006/03/30 12:26:45 | Loaded native library: libfnordservice.so
INFO   | 2006/03/30 12:26:45 | Calling native initialization method.
INFO   | 2006/03/30 12:26:45 | Inside native WrapperManager
initialization method
INFO   | 2006/03/30 12:26:45 | Java Version   : J2RE 1.5.0 IBM J9 2.3
AIX ppc-32 j9vmap3223ifx-20060310 (JIT enabled)
INFO   | 2006/03/30 12:26:45 | J9VM - 20060220_05389_bHdSMR
INFO   | 2006/03/30 12:26:45 | JIT  - 20060220_2133_r8
INFO   | 2006/03/30 12:26:45 | GC   - 20060214_AA
INFO   | 2006/03/30 12:26:45 | Java VM Vendor : IBM Corporation
INFO   | 2006/03/30 12:26:45 |
INFO   | 2006/03/30 12:26:45 | Wrapper (Version 3.1.2)
http://wrapper.tanukisoftware.org
INFO   | 2006/03/30 12:26:45 |
INFO   | 2006/03/30 12:26:45 |
WrapperManager.start(org.tanukisoftware.wrapper.WrapperSimpleApp@7d6c7d6c,
args["-c", "fnord"]) called by thread: main
INFO   | 2006/03/30 12:26:45 | Open socket to wrapper...
DEBUG  | 2006/03/30 12:26:46 | accepted a socket from 127.0.0.1 on port 365=
52
INFO   | 2006/03/30 12:26:46 | Opened Socket
INFO   | 2006/03/30 12:26:46 | Send a packet KEY : FyIsmbXc06V1kO_V
INFO   | 2006/03/30 12:26:46 |
handleSocket(Socket[addr=3Dlocalhost/127.0.0.1,port=3D32000,localport=3D365=
52])
DEBUG  | 2006/03/30 12:26:46 | read a packet KEY : FyIsmbXc06V1kO_V
DEBUG  | 2006/03/30 12:26:46 | Got key from JVM: FyIsmbXc06V1kO_V
DEBUG  | 2006/03/30 12:26:46 | send a packet LOW_LOG_LEVEL : 1
DEBUG  | 2006/03/30 12:26:46 | send a packet PING_TIMEOUT : 300
DEBUG  | 2006/03/30 12:26:46 | Start Application.
DEBUG  | 2006/03/30 12:26:46 | send a packet START : start
INFO   | 2006/03/30 12:26:46 | Received a packet LOW_LOG_LEVEL : 1
INFO   | 2006/03/30 12:26:46 | Wrapper Manager: LowLogLevel from Wrapper is=
 1
INFO   | 2006/03/30 12:26:46 | Received a packet PING_TIMEOUT : 300
INFO   | 2006/03/30 12:26:46 | Wrapper Manager: PingTimeout from
Wrapper is 300000
INFO   | 2006/03/30 12:26:46 | Received a packet START : start
INFO   | 2006/03/30 12:26:46 | calling listener.start()
INFO   | 2006/03/30 12:26:46 | WrapperSimpleApp: start(args)
INFO   | 2006/03/30 12:26:46 | WrapperSimpleApp: invoking main method
INFO   | 2006/03/30 12:26:46 | WrapperSimpleApp: main method completed
INFO   | 2006/03/30 12:26:46 | WrapperSimpleApp: start(args) end.=20
Main Completed=3Dtrue, exitCode=3Dnull
INFO   | 2006/03/30 12:26:46 | returned from listener.start()
INFO   | 2006/03/30 12:26:46 | Send a packet STARTED :
INFO   | 2006/03/30 12:26:46 | All non-daemon threads have stopped.  Exitin=
g.
INFO   | 2006/03/30 12:26:46 | WrapperManager.stop(0) called by
thread: Wrapper-Connection
INFO   | 2006/03/30 12:26:46 | Send a packet STOP : 0
DEBUG  | 2006/03/30 12:26:46 | read a packet STARTED :
DEBUG  | 2006/03/30 12:26:46 | JVM signalled that it was started.
DEBUG  | 2006/03/30 12:26:46 | read a packet STOP : 0
DEBUG  | 2006/03/30 12:26:46 | JVM requested a shutdown. (0)
DEBUG  | 2006/03/30 12:26:46 | wrapperStopProcess(0) called.
DEBUG  | 2006/03/30 12:26:46 | Sending stop signal to JVM
DEBUG  | 2006/03/30 12:26:46 | send a packet STOP : NULL
INFO   | 2006/03/30 12:26:47 | Thread, Wrapper-Connection, handling
the shutdown process.
INFO   | 2006/03/30 12:26:47 | calling listener.stop()
INFO   | 2006/03/30 12:26:47 | WrapperSimpleApp: stop(0)
INFO   | 2006/03/30 12:26:47 | returned from listener.stop()
INFO   | 2006/03/30 12:26:47 | Send a packet STOPPED : 0
DEBUG  | 2006/03/30 12:26:47 | read a packet STOPPED : 0
DEBUG  | 2006/03/30 12:26:47 | JVM signalled that it was stopped.
INFO   | 2006/03/30 12:26:47 | Closing socket.
DEBUG  | 2006/03/30 12:26:47 | socket read no code (closed?).
INFO   | 2006/03/30 12:26:48 | calling System.exit(0)
DEBUG  | 2006/03/30 12:26:48 | JVM process exited with a code of 0,
leaving the wrapper exit code set to 0.
DEBUG  | 2006/03/30 12:26:48 | JVM exited normally.
STATUS | 2006/03/30 12:26:48 | <-- Fnord World Domination Server 2006 Stopp=
ed

[fnordservice.conf]
#*********************************************************************
# FNORD World Domination Server Properties
#*********************************************************************

#*********************************************************************
# Title to use when running as a console.  It is also used in some messages=
.
# The default is set to "FNORD World Domination Server YYYY".
#*********************************************************************
fnord.console.title=3DFNORD World Domination Server 2006

# Log file to use for output logging; default is startup.log.
fnord.logfile=3D/home/clvrmnky/servers/3817aix/log/startup.log

# The name of the Java Native Library (JNI)
fnord.native_library=3Dfnordservice

#*********************************************************************
# fnord.java.additonal.<n>
#   Additional Java parameters to pass to Java when it is launched.
#   These are the parameters for the Java Virtual Machine.
#   Each element has a property name which starts with fnord.java.additiona=
l.
#   and ends with a unique number between 1 and 60. The numbering
#   and ordering in the file need not be continuous, but avoid duplicates.
#*********************************************************************
fnord.java.additional.1=3D-Xms512m
fnord.java.additional.2=3D-Xmx512m
fnord.java.additional.3=3D-Xbootclasspath/p:/home/clvrmnky/servers/3817aix/=
server/fnord/patch/patchfnordboot/classes:/home/clvrmnky/servers/3817aix/se=
rver/fnord/patch/patchfnordboot.jar:/home/clvrmnky/servers/3817aix/server/f=
nord/lib/boot/fnordboot.jar
fnord.java.additional.4=3D-Xss256k
fnord.java.additional.5=3D-Dprogram.name=3Drun.bat
fnord.java.additional.6=3D-Djboss.identity.domain=3DFNORD
fnord.java.additional.7=3D-Dfnord.safemode=3Dfalse
fnord.java.additional.8=3D-Dfnord.safemode.password=3D<secret>
fnord.java.additional.9=3D-Dfnord.unattended.startup=3Dtrue
fnord.java.additional.10=3D-Dfnord.master.key=3Dfake_master_key
fnord.java.additional.11=3D-Dfnord.dynamic.aop=3Dfalse
fnord.java.additional.12=3D-Dxmbean.metadata.xml.validate=3Dtrue
fnord.java.additional.13=3D-Dfnordis.neurondir=3D/home/clvrmnky/servers/381=
7aix/server/fnord/bin
fnord.java.additional.14=3D-Djboss.server.base.dir=3D/home/clvrmnky/servers=
/3817aix/server
fnord.java.additional.15=3D-Djboss.server.base.url=3Dfile:///home/clvrmnky/=
servers/3817aix/server
fnord.java.additional.17=3D-Djava.endorsed.dirs=3D/home/clvrmnky/servers/38=
17aix/server/lib/endorsed
fnord.java.additional.18=3D-Djava.awt.headless=3Dtrue
fnord.java.additional.19=3D-Dfnord.installDir=3D/home/clvrmnky/servers/3817=
aix
fnord.java.additional.20=3D-Dfnord.install=3Dserver
fnord.java.additional.21=3D-verbose:jni
fnord.java.additional.22=3D-Xcheck:jni
fnord.java.additional.23=3D-Dorg.tanukisoft.wrapper.WrapperSimpleApp.maxSta=
rtMainWait=3D300
#fnord.java.additional.21=3D-server
#fnord.java.additional.22=3D-XX:MaxPermSize=3D128m

# The command to use when launching a JVM.
fnord.java.command=3D/home/clvrmnky/servers/3817aix/jre/bin/java

#*********************************************************************
# Other Properties - modify these only under advice of FNORD services team
#*********************************************************************

#*********************************************************************
# fnord.java.classpath.<n> --- Java Classpath to use.
#   Each element has a property name which starts with fnord.java.classpath=
.
#   and ends with a unique number between 1 and 40. The numbering
#   and ordering in the file need not be continuous, but avoid duplicates.
#*********************************************************************
fnord.java.classpath.1=3D/home/clvrmnky/servers/3817aix/server/lib/fnordser=
vice.jar
fnord.java.classpath.2=3D/home/clvrmnky/servers/3817aix/server/bin/run.jar

# Class to execute when the service starts the application.
fnord.java.mainclass=3Dorg.tanukisoftware.wrapper.WrapperSimpleApp

#*********************************************************************
# fnord.app.parameter.<n>
#   Application parameters to pass to the application when it is launched.
#   Each element has a property name which starts with fnord.java.parameter=
.
#   and ends with a unique number between 1 and 40. The numbering
#   and ordering in the file need not be continuous, but avoid duplicates.
#*********************************************************************
fnord.app.parameter.1=3Dorg.jboss.Main
fnord.app.parameter.2=3D-c
fnord.app.parameter.3=3Dfnord

#*********************************************************************
# Format of output for console.  The format consists of the tokens
#       'L' for log level,
#       'D' for thread,
#       'T' for time, and
#       'M' for message.
# If the property is missing or commented out, then the default value 'M'
# will be used.  Setting the property to blank will disable console output.
# E.g. fnord.console.format=3DLTM
#       STATUS | 2005/11/16 11:00:25 | --> FNORD Started as Console
#=09STATUS | 2005/11/16 11:00:25 | Launching a JVM...
#*********************************************************************
fnord.console.format=3DM

#*********************************************************************
# Log level for console output.  Valid log levels include:
#       NONE for no output,
#       FATAL to only show fatal error messages,
#       ERROR to show all error messages,
#       STATUS to show all state changes (the default),
#       INFO to show all JVM output and informative messages, and
#       DEBUG to show detailed debug information.
# Default is STATUS. Note that at STATUS level, hitting CTRL-BREAK does
# not produce a stack trace when running in console mode.
#*********************************************************************
fnord.console.loglevel=3DDEBUG

#*********************************************************************
# Format of output for log file.  The format consists of the tokens
#       'L' for log level,
#       'D' for thread,
#       'T' for time, and
#       'M' for message.
# If the property is missing or commented out, then the default value 'LTM'
# will be used.  Setting the property to blank will disable file logging.
# E.g. fnord.logfile.format=3DLTM
#       STATUS | 2005/11/16 11:00:25 | --> FNORD Started as Console
#=09STATUS | 2005/11/16 11:00:25 | Launching a JVM...
#*********************************************************************
fnord.logfile.format=3DLTM

#*********************************************************************
# Log level for console output.  Valid log levels include:
#       NONE for no output,
#       FATAL to only show fatal error messages,
#       ERROR to show all error messages,
#       STATUS to show all state changes (the default),
#       INFO to show all JVM output and informative messages, and
#       DEBUG to show detailed debug information.
#*********************************************************************
fnord.logfile.loglevel=3DDEBUG

#*********************************************************************
# Maximum size that the log file will be allowed to grow to before the log
# is rolled.  Size is specified in bytes may be abbreviated with the 'k' (k=
b)
# or 'm' (mb) suffix.  The default value of 0 disables log rolling.
#*********************************************************************
fnord.logfile.maxsize=3D0

#*********************************************************************
# Maximum number of rolled log files which will be allowed before old files
# are deleted.  The default value of 0 implies no limit.
#*********************************************************************
fnord.logfile.maxfiles=3D0

#*********************************************************************
# Log Level for sys/event log output; default is NONE.
# Valid loglevels are the same as the other loglevel related properties.
#*********************************************************************
fnord.syslog.loglevel=3DNONE

#*********************************************************************
# fnord.java.library.path.<n> --- Java library path to use.
#   Each element has a property name which starts with fnord.java.library.p=
ath.
#   and ends with a unique number between 1 and 40. The numbering
#   and ordering in the file need not be continuous, but avoid duplicates.
#*********************************************************************
fnord.java.library.path.1=3D/home/clvrmnky/servers/3817aix/server/lib
fnord.java.library.path.2=3D/home/clvrmnky/servers/3817aix/server/fnord/bin

#*********************************************************************
# Number of seconds to allow between pinging the JVM and obtaining the
# response. 0 means never time out.
#*********************************************************************
fnord.ping.timeout=3D300

#*********************************************************************
# Number of seconds to wait between startup attempts.
#*********************************************************************
fnord.restart.delay=3D60

#*********************************************************************
# Number of startup attempts to try before giving up.
#*********************************************************************
fnord.max_failed_invocations=3D20

#*********************************************************************
# Database Specific Properties
#*********************************************************************
# DB2
#fnord.java.library.path.3=3D%DB2DIR%/java12
#fnord.java.library.path.4=3D%DB2DIR%/java
#fnord.java.library.path.5=3D%DB2DIR%/bin
#fnord.java.library.path.6=3D%DB2DIR%/lib
#fnord.java.classpath.3=3D%DB2DIR%/java12/db2java.zip
#fnord.java.classpath.4=3D%DB2DIR%/java/db2java.zip