#259 JVM Exits Unexpectedly in 64 bit Windows machine

v3.5.9
open
Service (34)
5
2011-07-04
2011-06-24
Eswar
No

Hi,
I am running a Java application as a service using Tanuki Wrapper (Trial Version (wrapper-windows-x86-64-3.5.9-pro) in

a 64 bit Windows 2003 Server with a 64 bit JVM . The Application starts well. But when i try to stop the service it takes a

long time and a dialog box appears with the following message "Could not stop the service on Local Computer. Error 1053:

The service did not respond to the start or control request in a timely fashion."
But the wrapper tries to stop the service as i gave wrapper.shutdown.timeout=300. After a few minutes I am getting the

following error in the wrapper log
"NOTICE | wrapper | 2011/06/21 11:43:53 |
NOTICE | wrapper | 2011/06/21 11:43:53 | --------------------------------------------------------------------
NOTICE | wrapper | 2011/06/21 11:43:53 | Thank you for your interest in the Java Service Wrapper.
NOTICE | wrapper | 2011/06/21 11:43:53 |
NOTICE | wrapper | 2011/06/21 11:43:53 | You are running with a full featured trial License Key which will
NOTICE | wrapper | 2011/06/21 11:43:53 | allow you to run or restart the Java Service Wrapper as many times
NOTICE | wrapper | 2011/06/21 11:43:53 | as you like until 2011/07/21. If the Wrapper is running at that
NOTICE | wrapper | 2011/06/21 11:43:53 | point, it will shutdown on its own after an additional 24 hours.
NOTICE | wrapper | 2011/06/21 11:43:53 |
NOTICE | wrapper | 2011/06/21 11:43:53 | The Java Service Wrapper requires a License Key to activate the
NOTICE | wrapper | 2011/06/21 11:43:53 | software. License Keys can be purchased on the Java Service Wrapper
NOTICE | wrapper | 2011/06/21 11:43:53 | web site:
NOTICE | wrapper | 2011/06/21 11:43:53 | http://wrapper.tanukisoftware.com/purchase
NOTICE | wrapper | 2011/06/21 11:43:53 |
NOTICE | wrapper | 2011/06/21 11:43:53 | This trial License Key in intended for testing and development and
NOTICE | wrapper | 2011/06/21 11:43:53 | should not be used as part of a production environment.
NOTICE | wrapper | 2011/06/21 11:43:53 | --------------------------------------------------------------------
NOTICE | wrapper | 2011/06/21 11:43:53 |
ERROR | wrapper | 2011/06/21 11:53:30 | Shutdown failed: Timed out waiting for signal from JVM.
ERROR | wrapper | 2011/06/21 11:53:31 | JVM did not exit on request, terminated"

I even increased the wrapper.shutdown.timeout to 1800 but still i am getting the above error.

The Same Application runs fine in a 32 bit Windows machine with a 32 bit JVM and with wrapper.shutdown.timeout=300.
This issue comes in a 64 bit Windows machine only.
Please help me to solve this issue.

Discussion

  • Eswar

    Eswar - 2011-06-24
    • priority: 5 --> 9
     
  • Leif Mortenson

    Leif Mortenson - 2011-06-26

    Eswar,
    Could you please set the wrapper.debug=TRUE and wrapper.request_thread_dump_on_failed_jvm_exit=TRUE properties, then rerun your test? I would need to see the full output of your log file to say what is happening. It should be fine to set the shutdown timeout back to its default.

    The Wrapper shutdown works by calling System.exit. The JVM then launches any registered shutdown hooks. It will NOT shutdown until all of those shutdown hooks have completed. This is correct operation.

    Most likely, one of the shutdown hooks is failing to exit. The thread dump requested before the JVM is killed will show us which threads are still active.

    Cheers,
    Leif

     
  • Leif Mortenson

    Leif Mortenson - 2011-06-26
    • assigned_to: nobody --> mortenson
    • priority: 9 --> 5
     
  • Eswar

    Eswar - 2011-06-27

    Wrapper Log for 64 bit Windows Server

     
  • Eswar

    Eswar - 2011-06-27

    Hi Lief,
    Thanks for your quick response. As per your comment i have made those two parameters as true. I have attached the recent log file (aftwrapper.log).
    Please check the log and help me to solve this issue.
    Thanks,
    Eswar

     
  • Eswar

    Eswar - 2011-06-29
    • priority: 5 --> 9
     
  • Leif Mortenson

    Leif Mortenson - 2011-06-29
    • priority: 9 --> 5
     
  • Leif Mortenson

    Leif Mortenson - 2011-06-29

    Eswar,
    I can see from your log that the Wrapper received the notification to start the shutdown and then called your WrapperListener implementation here:
    INFO | jvm 1 | 2011/06/27 06:11:09 | WrapperManager Debug: calling listener.stop()

    That method never returns though which is why the Wrapper eventually times out and kills the JVM.

    It looks like your WrapperListener implementation class is called com.rbsplc.afta.management.main.AftAdapterMainService.

    You can see that it is blocking "waiting to lock <0x00000000e58b0e70> (a java.lang.Object)" inside of the com.rbsplc.afta.adapter.argon.handler.ArgonFileReceiveHandler.terminateHandler method.

    INFO | jvm 1 | 2011/06/27 06:16:42 | "WrapperListener_stop_runner" prio=6 tid=0x00000000054b5800 nid=0x5a0 waiting for monitor entry [0x00000000062bf000]
    INFO | jvm 1 | 2011/06/27 06:16:42 | java.lang.Thread.State: BLOCKED (on object monitor)
    INFO | jvm 1 | 2011/06/27 06:16:42 | at com.rbsplc.afta.adapter.argon.handler.ArgonFileReceiveHandler.terminateHandler(ArgonFileReceiveHandler.java:214)
    INFO | jvm 1 | 2011/06/27 06:16:42 | - waiting to lock <0x00000000e58b0e70> (a java.lang.Object)
    INFO | jvm 1 | 2011/06/27 06:16:42 | at com.rbsplc.afta.adapter.argon.handler.ArgonClientHandler.terminateHandler(ArgonClientHandler.java:126)
    INFO | jvm 1 | 2011/06/27 06:16:42 | at com.rbsplc.afta.adapter.argon.ArgonAdapter.terminate(ArgonAdapter.java:92)
    INFO | jvm 1 | 2011/06/27 06:16:42 | at com.rbsplc.afta.management.main.AftAdapterMainService.terminateApplicationHookups(AftAdapterMainService.java:245)
    INFO | jvm 1 | 2011/06/27 06:16:42 | at com.rbsplc.afta.management.main.AftAdapterMainService.stop(AftAdapterMainService.java:173)
    INFO | jvm 1 | 2011/06/27 06:16:42 | at org.tanukisoftware.wrapper.WrapperManager$13.run(WrapperManager.java:4320)

    The object that you are waiting for is being locked in the thread ArgonReceiver_eqnr_sink_feed here:

    INFO | jvm 1 | 2011/06/27 06:16:42 | "ArgonReceiver_eqnr_sink_feed" prio=6 tid=0x000000000553f800 nid=0x130 in Object.wait() [0x000000000afbf000]
    INFO | jvm 1 | 2011/06/27 06:16:42 | java.lang.Thread.State: WAITING (on object monitor)
    INFO | jvm 1 | 2011/06/27 06:16:42 | at java.lang.Object.wait(Native Method)
    INFO | jvm 1 | 2011/06/27 06:16:42 | at java.lang.Object.wait(Object.java:485)
    INFO | jvm 1 | 2011/06/27 06:16:42 | at com.rbsfm.argon.common.util.EventLatch.waitForEvent(EventLatch.java:64)
    INFO | jvm 1 | 2011/06/27 06:16:42 | - locked <0x00000000db41acc8> (a java.lang.Object)
    INFO | jvm 1 | 2011/06/27 06:16:42 | at com.rbsfm.argon.client.cga.Receiver$ReceiverThread.waitForMessageorTimeOut(Receiver.java:420)
    INFO | jvm 1 | 2011/06/27 06:16:42 | at com.rbsfm.argon.client.cga.Receiver.receive(Receiver.java:1069)
    INFO | jvm 1 | 2011/06/27 06:16:42 | at com.rbsplc.afta.adapter.argon.handler.ArgonFileReceiveHandler.receiveMessage(ArgonFileReceiveHandler.java:601)
    INFO | jvm 1 | 2011/06/27 06:16:42 | - locked <0x00000000e58b0e70> (a java.lang.Object)
    INFO | jvm 1 | 2011/06/27 06:16:42 | at com.rbsplc.afta.adapter.argon.handler.ArgonFileReceiveHandler.access$3(ArgonFileReceiveHandler.java:578)
    INFO | jvm 1 | 2011/06/27 06:16:42 | at com.rbsplc.afta.adapter.argon.handler.ArgonFileReceiveHandler$ArgonReceiveExecutor.run(ArgonFileReceiveHandler.java:672)
    INFO | jvm 1 | 2011/06/27 06:16:42 | at java.lang.Thread.run(Unknown Source)

    The Wrapper waited 333 seconds before killing the JVM because you have the long shutdown timeout. If you do not think that more time will fix the problem then you will need to go in and review your code to find out why that object is not being released.

    I hope this helps.

    Cheers,
    Leif

     
  • Eswar

    Eswar - 2011-07-04

    Hi Lief,
    Thanks for your reply. But we are using the same code in the 32 bit Windows server and 64 bit Windows server. Then how come the Application runs successfully in 32 bit Windows Server and not stopping properly in 64 bit Windows server. The only change we did is included your 64 bit wrapper instead of 32 bit wrapper. So can you tell us what changes are made in 64 bit wrapper, so that we can do corresponding changes in our Application code.

     
  • Eswar

    Eswar - 2011-07-04
    • priority: 5 --> 9
     
  • Leif Mortenson

    Leif Mortenson - 2011-07-04

    Eswar,
    The stack trace and explanation that I sent you last time shows where in your code the stop thread is blocking. Unfortunately, it is not possible to help further without seeing all of the related source code.

    There are no differences between the 32 and 64-bit versions which I can think of that would affect this, but your user code may be encountering slightly different data which could be affecting the timing.

    Is it possible for you to look at the source of the methods that I pointed out and see what is happening in the source code? I do not see any evidence that the issue is directly related to the Wrapper other than the fact that the locked object is preventing the Wrapper from shutting down cleanly. The Wrapper appears to be working as designed in this case however as it is correctly waiting for your application to shutdown cleanly.

    Cheers,
    Leif

     
  • Leif Mortenson

    Leif Mortenson - 2011-07-04
    • priority: 9 --> 5
     

Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks