Work at SourceForge, help us to make it a better place! We have an immediate need for a Support Technician in our San Francisco or Denver office.

Close

#151 SegFault "Wrapper-Control-Event-Monitor" Linux 64 Bit

v3.2.3
closed-accepted
JNI (8)
7
2010-03-22
2006-12-12
Peter Jodeleit
No

We had some hs_err_pidXX.files caused by "Wrapper-Control-Event-Monitor" under linux 64 bit.

xxx$ java -version
java version "1.5.0_10"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_10-b03)
Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_10-b03, mixed mode)

xxxx$ grep "Current thread" hs*.log
hs_err_pid10499.log:Current thread (0x00002aaaaf162be0): JavaThread "Wrapper-Control-Event-Monitor" daemon [_thread_in_native, id=10521]
hs_err_pid1672.log:Current thread (0x00002aaaae459b90): JavaThread "Wrapper-Control-Event-Monitor" daemon [_thread_in_native, id=1699]
hs_err_pid19540.log:Current thread (0x00002aaaae7c0a00): JavaThread "Wrapper-Control-Event-Monitor" daemon [_thread_in_native, id=19572]
hs_err_pid21579.log:Current thread (0x00002aaab082a3e0): JavaThread "Wrapper-Control-Event-Monitor" daemon [_thread_in_native, id=21601]
hs_err_pid24642.log:Current thread (0x00002aaaae0bc650): JavaThread "Wrapper-Control-Event-Monitor" daemon [_thread_in_native, id=24670]
hs_err_pid25683.log:Current thread (0x00002aaaaeed6070): JavaThread "Wrapper-Control-Event-Monitor" daemon [_thread_in_native, id=25707]
hs_err_pid2787.log:Current thread (0x00002aaaadb8d2e0): JavaThread "Wrapper-Control-Event-Monitor" daemon [_thread_in_native, id=2844]
hs_err_pid28643.log:Current thread (0x00002aaab0308070): JavaThread "Wrapper-Control-Event-Monitor" daemon [_thread_in_native, id=28688]
hs_err_pid30324.log:Current thread (0x00002aaaafe473e0): JavaThread "Wrapper-Control-Event-Monitor" daemon [_thread_in_native, id=30354]
hs_err_pid31963.log:Current thread (0x00002aaaaabd1910): JavaThread "Wrapper-Control-Event-Monitor" daemon [_thread_in_native, id=31985]
hs_err_pid3321.log:Current thread (0x00002aaaadc990a0): JavaThread "Wrapper-Control-Event-Monitor" daemon [_thread_in_native, id=3354]
hs_err_pid4197.log:Current thread (0x00002aaaada604f0): JavaThread "Wrapper-Control-Event-Monitor" daemon [_thread_in_native, id=4226]
hs_err_pid4814.log:Current thread (0x00002aaaadc489b0): JavaThread "Wrapper-Control-Event-Monitor" daemon [_thread_in_native, id=4849]
hs_err_pid5242.log:Current thread (0x00002aaaafa1c1d0): JavaThread "Wrapper-Control-Event-Monitor" daemon [_thread_in_native, id=5268]
hs_err_pid8328.log:Current thread (0x00002aaaaee29da0): JavaThread "Wrapper-Control-Event-Monitor" daemon [_thread_in_native, id=8359]
hs_err_pid9089.log:Current thread (0x00002aaaae047e70): JavaThread "Wrapper-Control-Event-Monitor" daemon [_thread_in_native, id=9125]
hs_err_pid9503.log:Current thread (0x00002aaaaf508480): JavaThread "Wrapper-Control-Event-Monitor" daemon [_thread_in_native, id=9537]

xxxx$ grep "Java VM" hs*.log
hs_err_pid10499.log:# Java VM: Java HotSpot(TM) 64-Bit Server VM (1.5.0_10-b03 mixed mode)
hs_err_pid1672.log:# Java VM: Java HotSpot(TM) 64-Bit Server VM (1.5.0_06-b05 mixed mode)
hs_err_pid19540.log:# Java VM: Java HotSpot(TM) 64-Bit Server VM (1.5.0_09-b03 mixed mode)
hs_err_pid21579.log:# Java VM: Java HotSpot(TM) 64-Bit Server VM (1.5.0_06-b05 mixed mode)
hs_err_pid24642.log:# Java VM: Java HotSpot(TM) 64-Bit Server VM (1.5.0_06-b05 mixed mode)
hs_err_pid25683.log:# Java VM: Java HotSpot(TM) 64-Bit Server VM (1.5.0_10-b03 mixed mode)
hs_err_pid2787.log:# Java VM: Java HotSpot(TM) 64-Bit Server VM (1.5.0_06-b05 mixed mode)
hs_err_pid28643.log:# Java VM: Java HotSpot(TM) 64-Bit Server VM (1.5.0_10-b03 mixed mode)
hs_err_pid30324.log:# Java VM: Java HotSpot(TM) 64-Bit Server VM (1.5.0_09-b03 mixed mode)
hs_err_pid31963.log:# Java VM: Java HotSpot(TM) 64-Bit Server VM (1.5.0_06-b05 mixed mode)
hs_err_pid3321.log:# Java VM: Java HotSpot(TM) 64-Bit Server VM (1.5.0_09-b03 mixed mode)
hs_err_pid4197.log:# Java VM: Java HotSpot(TM) 64-Bit Server VM (1.5.0_10-b03 mixed mode)
hs_err_pid4814.log:# Java VM: Java HotSpot(TM) 64-Bit Server VM (1.5.0_10-b03 mixed mode)
hs_err_pid5242.log:# Java VM: Java HotSpot(TM) 64-Bit Server VM (1.5.0_06-b05 mixed mode)
hs_err_pid8328.log:# Java VM: Java HotSpot(TM) 64-Bit Server VM (1.5.0_09-b03 mixed mode)
hs_err_pid9089.log:# Java VM: Java HotSpot(TM) 64-Bit Server VM (1.5.0_06-b05 mixed mode)
hs_err_pid9503.log:# Java VM: Java HotSpot(TM) 64-Bit Server VM (1.5.0_06-b05 mixed mode)

Content excerpt of hs_err_pidXX.file:

#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# SIGSEGV (0xb) at pc=0x00002aaab1701950, pid=10499, tid=1096710512
#
# Java VM: Java HotSpot(TM) 64-Bit Server VM (1.5.0_10-b03 mixed mode)
# Problematic frame:
# C 0x00002aaab1701950
#

--------------- T H R E A D ---------------

Current thread (0x00002aaaaf162be0): JavaThread "Wrapper-Control-Event-Monitor" daemon [_thread_in_native, id=10521]

siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x00002aaab1701950

Registers:
RAX=0x00000002b656a0f8, RBX=0x00002b381a277f50, RCX=0x20c49ba5e353f7cf, RDX=0x0000000000000050
RSP=0x00000000415e7278, RBP=0x0000000080043549, RSI=0x00000000415e7280, RDI=0x00002aaaaf162d30
R8 =0x0000000000000000, R9 =0x00002aaaaf162be0, R10=0x00002aaab1701950, R11=0x00002aaaadd61c50
R12=0x00000000415e7280, R13=0x0000000000000000, R14=0x00000000415e7418, R15=0x00002aaaaf162be0
RIP=0x00002aaab1701950, EFL=0x0000000000010206, CSGSFS=0x0000000000000033, ERR=0x0000000000000014
TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00000000415e7278)
0x00000000415e7278: 00002b381999feb9 00002b381d1992d0
0x00000000415e7288: 0000000080043549 0000000000000000
0x00000000415e7298: 00000000415e7418 00002aaaaf162be0
0x00000000415e72a8: 00000000000d6da9 0000000080043548
0x00000000415e72b8: 00002b3819a96d20 00002aaaaf162be0
0x00000000415e72c8: 36526baf36526baf 8004354900000000
0x00000000415e72d8: 00002b3800000000 0000000000000064
0x00000000415e72e8: 00002b3819536020 00000000415e73b8
0x00000000415e72f8: 00002b3819530a4e 00002aaaaf162be0
0x00000000415e7308: 0000000000000000 00002aaaaf162be0
0x00000000415e7318: 00000000415e7318 00002aaaaf162be0
0x00000000415e7328: 00000000415e73b8 00002b381d1a4ca2
0x00000000415e7338: 00000000415e7418 00002b3825050c90
0x00000000415e7348: 00002b381d1a4ca2 00000000415e7418
0x00000000415e7358: 00002b38247cc0d0 00000000415e73b8
0x00000000415e7368: 00002b3819571f60 0000000000000064
0x00000000415e7378: 00002b3819536020 00000000415e7380
0x00000000415e7388: 00002b381d1a4ca2 00000000415e7418
0x00000000415e7398: 00002b381d1b2f00 00002b381f3b7d90
0x00000000415e73a8: 00002b381d1a4eb0 00000000415e7418
0x00000000415e73b8: 00000000415e7480 00002b381952432d
0x00000000415e73c8: 0000000000000000 0000000000000000
0x00000000415e73d8: 0000000000000000 0000000000000064
0x00000000415e73e8: 0000000036523fc8 0000000080003c19
0x00000000415e73f8: 0000000000000000 0000000000000000
0x00000000415e7408: 0000000036523fc8 00002b3824bdd148
0x00000000415e7418: 00002b385dc8dbf0 0000000000001fa0
0x00000000415e7428: 00000000415e74c0 00000000415e7650
0x00000000415e7438: 00002aaaaf162be0 00002b3815533678
0x00000000415e7448: 00002b38195242a9 00000000415e74f0
0x00000000415e7458: 00000000415e7708 000000000000000a
0x00000000415e7468: 00002b381d1a4eb0 00002b381952eb60

Instructions: (pc=0x00002aaab1701950)
0x00002aaab1701940:
[error occurred during error reporting, step 100, id 0xb]

Stack: [0x00000000414e8000,0x00000000415e8000), sp=0x00000000415e7278, free space=1020k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C 0x00002aaab1701950

[error occurred during error reporting, step 120, id 0xb]

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J org.tanukisoftware.wrapper.WrapperManager.nativeGetControlEvent()I
J org.tanukisoftware.wrapper.WrapperManager$3.run()V
v ~OSRAdapter
v ~StubRoutines::call_stub

Discussion

  • Leif Mortenson
    Leif Mortenson
    2007-01-06

    Logged In: YES
    user_id=228081
    Originator: NO

    Peter,
    The signal handling code was storing a signal into a volatile variable in one thread and then simply reading it from another. This read/write was always safe in 32-bit versions as the write of a single 32-bit word was atomic.

    It appears that that this was not true with 64-bit systems. I have added full mutex support to synchronize access to that code. I would like to get this change into the next release. Could you download the following prerelease version and test out this fix?

    http://wrapper.tanukisoftware.org/tmp/3.2.4-a/wrapper-linux-x86-64-3.2.4-a.tar.gz

    Also, how easy is it to reproduce this problem with the version you are using? (v3.2.3?)

    Cheers,
    Leif

     
  • Leif Mortenson
    Leif Mortenson
    2007-01-06

    • milestone: --> v3.2.3
    • assigned_to: nobody --> mortenson
     
  • Leif Mortenson
    Leif Mortenson
    2007-01-06

    • labels: --> JNI
    • priority: 5 --> 7
    • status: open --> open-accepted
     
  • Peter Jodeleit
    Peter Jodeleit
    2007-01-08

    Logged In: YES
    user_id=951561
    Originator: YES

    We used v3.2.1, actually we changed to v3.2.3. I'll give a feedback about the behaviour of this version asap.

    Seit Freitag 3.2.3 also die letzte Stable. Damit sollte man das Verhalten
    beim Stoppen auch noch mal überprüfen, vorher war es nämlich 3.2.1.

    > how easy is it to reproduce this problem with the version you are using?

    I would call it "easy". It's an test environment where we restart the application about once a day. The dumps were produced in about 5 weeks, so about 70% "stop" signals failed.

    Btw: If a read depends on a former write this must be always synchronized (e.g. "i++" must be synchronzied, even if "int i" is defined as "volatile"). Our test server is an smp environment..

     
  • Leif Mortenson
    Leif Mortenson
    2007-01-09

    Logged In: YES
    user_id=228081
    Originator: NO

    That wasn't all in English.... :-)

    Ok. Please test it out and let me know how it looks. That code is now all fully synchronized so it should be Ok.

    As to the old method. I realize that things like i++ must be synchronized. That is because multiple threads are both reading and writing.

    In the case of this code however. The signal thread is writing to variable A and reading from variable B. While a thread from the JVM is reading from variable A and writing to variable B. If the JVM thread reads an old value, it is fine and will be picked up on the next pass. This has been working in the wrapper for several years without any problems. Under 64-bit however, it appears that this does not function the same for some reason. As I understand it. The memory store operation for this 8-bit variable should be a 32-bit memory write as the variables will all be 32-bit aligned in memory. On the 64-bit system, that same write should be in a 64-bit variable and thus be a 64-bit write. It may be an alignment issue where the variable is sharing a 64-bit memory location with other data that must be read before the full 64-bits can be written. If so, that would definitely cause problems.

    Anyway, it should all be safe now. There is no place else in the code where I was doing anything like this.

    Cheers,
    Leif

     
  • Peter Jodeleit
    Peter Jodeleit
    2007-11-08

    Logged In: YES
    user_id=951561
    Originator: YES

    Works for me, issue could be closed..

     
  • Leif Mortenson
    Leif Mortenson
    2010-03-22

    This was fixed back in version 3.3.0.

     
  • Leif Mortenson
    Leif Mortenson
    2010-03-22

    • status: open-accepted --> closed-accepted