SourceForge has been redesigned. Learn more.
Close

#13 Potential crash when starting concurrently

411.20130714
closed-fixed
C++ (2)
5
2013-10-08
2012-11-29
Erik Duijs
No

When concurrently starting a REXX script from java, BSF4ooRexx seems prone to crashing (taking the JVM down with it).

See the attached test program to reproduce and a crash report. It doesn't consistently crash 100% of the time, but usually it does. Raising the THREADS constant seems to increase the likelyhood of the crash.
The vulnerability seems to be mainly while starting the REXX script: If it does start successfully, it seems to keep running as it should.

Discussion

  • Erik Duijs

    Erik Duijs - 2012-11-29

    JVM crash report

     
  • Rony G. Flatscher

    Thanks, Erik, will look into it sometimes in the next week. So anyone beating me, is highly welcome! ;)

     
  • Rony G. Flatscher

    • labels: --> C++
    • assigned_to: nobody --> orexx
    • milestone: --> Next Release
    • status: open --> pending-fixed
     
  • Erik Duijs

    Erik Duijs - 2013-01-08

    I've tested it with the 20130107 beta version, and unfortunately I could still crash the JVM with the TestConcurrencyCrash.java test program (although perhaps it happens less frequently).

    This is in the JVM crash log:
    #
    # A fatal error has been detected by the Java Runtime Environment:
    #
    # EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x6d95744a, pid=4620, tid=1672
    #
    # JRE version: 6.0_33-b05
    # Java VM: Java HotSpot(TM) Client VM (20.8-b03 mixed mode, sharing windows-x86 )
    # Problematic frame:
    # V [jvm.dll+0x9744a]
    #
    # If you would like to submit a bug report, please visit:
    # http://java.sun.com/webapps/bugreport/crash.jsp
    #

    --------------- T H R E A D ---------------

    Current thread (0x02b8f000): JavaThread "Thread-1" [_thread_in_vm, id=1672, stack(0x02ed0000,0x02f20000)]

    siginfo: ExceptionCode=0xc0000005, reading address 0x00000000

    Registers:
    EAX=0x00000000, EBX=0x02b8f128, ECX=0x00000006, EDX=0x02ba2c68
    ESP=0x02f1f484, EBP=0x02f1f4cc, ESI=0x02b8f000, EDI=0x00000000
    EIP=0x6d95744a, EFLAGS=0x00010246

    Top of Stack: (sp=0x02f1f484)
    0x02f1f484: 00000000 02b8f128 000a6b80 02f1f9c0
    0x02f1f494: 02b8f128 000a6b80 000a6000 000a6000
    0x02f1f4a4: 02f1f490 ffffffff 02f1f4ec 7c839ab0
    0x02f1f4b4: 7c809f08 ffffffff 02b8f000 00000000
    0x02f1f4c4: 000a6b80 7c810068 00000048 100012de
    0x02f1f4d4: 02b8f128 02ba2c68 00000000 02f1f4f4
    0x02f1f4e4: 10002788 02b8f128 02ba2c68 00000000
    0x02f1f4f4: 02f1f9c0 02b8f128 00000048 000a6b80

    Instructions: (pc=0x6d95744a)
    0x6d95742a: ff 8b 46 04 83 c4 08 85 c0 89 75 f0 c7 45 f4 00
    0x6d95743a: 00 00 00 74 08 8d 4d f0 e8 49 a2 09 00 8b 7d 10
    0x6d95744a: 8b 07 c7 45 e0 0c 00 00 00 8b 48 08 0f b7 51 2a
    0x6d95745a: 8b 40 0c 8b 4c 90 28 51 56 8d 4d c4 e8 05 8c 07

    Register to memory mapping:

    EAX=0x00000000 is an unknown value
    EBX=0x02b8f128 is an unknown value
    ECX=0x00000006 is an unknown value
    EDX=0x02ba2c68 is an unknown value
    ESP=0x02f1f484 is pointing into the stack for thread: 0x02b8f000
    EBP=0x02f1f4cc is pointing into the stack for thread: 0x02b8f000
    ESI=0x02b8f000 is a thread
    EDI=0x00000000 is an unknown value

    Stack: [0x02ed0000,0x02f20000], sp=0x02f1f484, free space=317k
    Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
    V [jvm.dll+0x9744a]

    Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
    j org.rexxla.bsf.engines.rexx.RexxAndJava.jniRexxCreateInterpreterInstance([Ljava/lang/Object;)Ljava/lang/String;+0
    j org.rexxla.bsf.engines.rexx.RexxAndJava.createRexxInterpreterInstance(Lorg/rexxla/bsf/engines/rexx/RexxConfiguration;)Ljava/lang/String;+65
    j org.rexxla.bsf.engines.rexx.RexxEngine.eval(Ljava/lang/String;IILjava/lang/Object;)Ljava/lang/Object;+111
    j org.apache.bsf.util.BSFEngineImpl.exec(Ljava/lang/String;IILjava/lang/Object;)V+6
    j org.rexxla.bsf.bugreport.TestConcurrencyCrash$1.run()V+33
    j java.lang.Thread.run()V+11
    v ~StubRoutines::call_stub

    --------------- P R O C E S S ---------------

    Java Threads: ( => current thread )
    0x003c6c00 JavaThread "DestroyJavaVM" [_thread_blocked, id=5824, stack(0x008c0000,0x00910000)]
    0x02bc5000 JavaThread "Thread-15" [_thread_in_native, id=1740, stack(0x03330000,0x03380000)]
    0x02bc3000 JavaThread "Thread-14" [_thread_in_native, id=3836, stack(0x032e0000,0x03330000)]
    0x02bc1400 JavaThread "Thread-13" [_thread_in_native, id=5196, stack(0x03290000,0x032e0000)]
    0x02bbd000 JavaThread "Thread-12" [_thread_in_native, id=5404, stack(0x03240000,0x03290000)]
    0x02bae000 JavaThread "Thread-11" [_thread_in_native, id=5796, stack(0x031f0000,0x03240000)]
    0x02bac800 JavaThread "Thread-10" [_thread_in_native, id=4008, stack(0x031a0000,0x031f0000)]
    0x02bab000 JavaThread "Thread-9" [_thread_in_native, id=3840, stack(0x03150000,0x031a0000)]
    0x02baa000 JavaThread "Thread-8" [_thread_in_native, id=4056, stack(0x03100000,0x03150000)]
    0x02ba5000 JavaThread "Thread-7" [_thread_in_native, id=4176, stack(0x030b0000,0x03100000)]
    0x02ba3400 JavaThread "Thread-6" [_thread_in_native, id=476, stack(0x03060000,0x030b0000)]
    0x02ba1800 JavaThread "Thread-5" [_thread_in_native, id=1216, stack(0x03010000,0x03060000)]
    0x02ba0400 JavaThread "Thread-4" [_thread_in_native, id=5992, stack(0x02fc0000,0x03010000)]
    0x02b9ec00 JavaThread "Thread-3" [_thread_in_native, id=5988, stack(0x02f70000,0x02fc0000)]
    0x02b90400 JavaThread "Thread-2" [_thread_in_native, id=4540, stack(0x02f20000,0x02f70000)]
    =>0x02b8f000 JavaThread "Thread-1" [_thread_in_vm, id=1672, stack(0x02ed0000,0x02f20000)]
    0x02b9cc00 JavaThread "Thread-0" [_thread_in_native, id=948, stack(0x02e80000,0x02ed0000)]
    0x02b71c00 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=1172, stack(0x02de0000,0x02e30000)]
    0x02b63000 JavaThread "C1 CompilerThread0" daemon [_thread_blocked, id=1132, stack(0x02d90000,0x02de0000)]
    0x02b61400 JavaThread "Attach Listener" daemon [_thread_blocked, id=3084, stack(0x02d40000,0x02d90000)]
    0x02b60000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=5372, stack(0x02cf0000,0x02d40000)]
    0x02b5b800 JavaThread "Finalizer" daemon [_thread_blocked, id=2536, stack(0x02ca0000,0x02cf0000)]
    0x02b56c00 JavaThread "Reference Handler" daemon [_thread_blocked, id=5800, stack(0x02c50000,0x02ca0000)]

    Other Threads:
    0x02b1ac00 VMThread [stack: 0x02c00000,0x02c50000] [id=5952]
    0x02b7d400 WatcherThread [stack: 0x02e30000,0x02e80000] [id=6036]

    VM state:not at safepoint (normal execution)

    VM Mutex/Monitor currently owned by a thread: None

    Heap
    def new generation total 4928K, used 2564K [0x22990000, 0x22ee0000, 0x27ee0000)
    eden space 4416K, 58% used [0x22990000, 0x22c11180, 0x22de0000)
    from space 512K, 0% used [0x22de0000, 0x22de0000, 0x22e60000)
    to space 512K, 0% used [0x22e60000, 0x22e60000, 0x22ee0000)
    tenured generation total 10944K, used 0K [0x27ee0000, 0x28990000, 0x32990000)
    the space 10944K, 0% used [0x27ee0000, 0x27ee0000, 0x27ee0200, 0x28990000)
    compacting perm gen total 12288K, used 272K [0x32990000, 0x33590000, 0x36990000)
    the space 12288K, 2% used [0x32990000, 0x329d4140, 0x329d4200, 0x33590000)
    ro space 10240K, 54% used [0x36990000, 0x36f0f260, 0x36f0f400, 0x37390000)
    rw space 12288K, 55% used [0x37390000, 0x37a33000, 0x37a33000, 0x37f90000)

    Code Cache [0x00920000, 0x009a8000, 0x02920000)
    total_blobs=181 nmethods=38 adapters=79 free_code_cache=33020032 largest_free_block=0

    Dynamic libraries:
    0x00400000 - 0x00425000 C:\Program Files\Java\jdk1.6.0_33\jre\bin\javaw.exe
    0x7c900000 - 0x7c9b2000 C:\WINDOWS\system32\ntdll.dll
    0x7c800000 - 0x7c8f6000 C:\WINDOWS\system32\kernel32.dll
    0x77dd0000 - 0x77e6b000 C:\WINDOWS\system32\ADVAPI32.dll
    0x77e70000 - 0x77f03000 C:\WINDOWS\system32\RPCRT4.dll
    0x77fe0000 - 0x77ff1000 C:\WINDOWS\system32\Secur32.dll
    0x7e410000 - 0x7e4a1000 C:\WINDOWS\system32\USER32.dll
    0x77f10000 - 0x77f59000 C:\WINDOWS\system32\GDI32.dll
    0x76390000 - 0x763ad000 C:\WINDOWS\system32\IMM32.DLL
    0x629c0000 - 0x629c9000 C:\WINDOWS\system32\LPK.DLL
    0x74d90000 - 0x74dfb000 C:\WINDOWS\system32\USP10.dll
    0x641f0000 - 0x641fc000 C:\PROGRA~1\NetInst\NiAMH.dll
    0x7c340000 - 0x7c396000 C:\Program Files\Java\jdk1.6.0_33\jre\bin\msvcr71.dll
    0x6d8c0000 - 0x6db6f000 C:\Program Files\Java\jdk1.6.0_33\jre\bin\client\jvm.dll
    0x76b40000 - 0x76b6d000 C:\WINDOWS\system32\WINMM.dll
    0x6d870000 - 0x6d87c000 C:\Program Files\Java\jdk1.6.0_33\jre\bin\verify.dll
    0x6d3d0000 - 0x6d3ef000 C:\Program Files\Java\jdk1.6.0_33\jre\bin\java.dll
    0x6d8b0000 - 0x6d8bf000 C:\Program Files\Java\jdk1.6.0_33\jre\bin\zip.dll
    0x10000000 - 0x10022000 C:\Program Files\BSF4ooRexx\BSF4ooRexx.dll
    0x03580000 - 0x03652000 C:\Program Files\ooRexx\rexx.dll
    0x773d0000 - 0x774d3000 C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.2600.6028_x-ww_61e65202\COMCTL32.dll
    0x77c10000 - 0x77c68000 C:\WINDOWS\system32\msvcrt.dll
    0x77f60000 - 0x77fd6000 C:\WINDOWS\system32\SHLWAPI.dll
    0x03660000 - 0x03685000 C:\Program Files\ooRexx\REXXAPI.dll
    0x71ad0000 - 0x71ad9000 C:\WINDOWS\system32\WSOCK32.dll
    0x71ab0000 - 0x71ac7000 C:\WINDOWS\system32\WS2_32.dll
    0x71aa0000 - 0x71aa8000 C:\WINDOWS\system32\WS2HELP.dll
    0x6d0b0000 - 0x6d1fc000 C:\Program Files\Java\jdk1.6.0_33\jre\bin\awt.dll
    0x73000000 - 0x73026000 C:\WINDOWS\system32\WINSPOOL.DRV
    0x774e0000 - 0x7761e000 C:\WINDOWS\system32\ole32.dll
    0x76bf0000 - 0x76bfb000 C:\WINDOWS\system32\PSAPI.DLL
    0x6b1f0000 - 0x6b214000 C:\Program Files\ooRexx\OREXXOLE.dll
    0x77120000 - 0x771ab000 C:\WINDOWS\system32\OLEAUT32.dll
    0x6b120000 - 0x6b155000 C:\Program Files\ooRexx\rexxutil.dll
    0x71a50000 - 0x71a8f000 C:\WINDOWS\system32\mswsock.dll
    0x662b0000 - 0x66308000 C:\WINDOWS\system32\hnetcfg.dll
    0x71a90000 - 0x71a98000 C:\WINDOWS\System32\wshtcpip.dll
    0x76f20000 - 0x76f47000 C:\WINDOWS\system32\DNSAPI.dll

    VM Arguments:
    jvm_args: -Dfile.encoding=Cp1252
    java_command: org.rexxla.bsf.bugreport.TestConcurrencyCrash
    Launcher Type: SUN_STANDARD

    Environment Variables:
    CLASSPATH=C:\Program Files\BSF4ooRexx\bsf-rexx-engine.jar;C:\Program Files\BSF4ooRexx\bsf-v400-20090910.jar;.
    PATH=C:/Program Files/Java/Jre6/bin/client;C:/Program Files/Java/Jre6/bin;C:/Program Files/Java/Jre6/lib/i386;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\IBM\Personal Communications;C:\Program Files\IBM\Trace Facility;C:\Program Files\ooRexx;C:\WINDOWS\system32\WindowsPowerShell\v1.0;C:\Epoprogs\UserEXE;C:\Program Files\Gs\Gs9.02\Bin;C:\Program Files\Java\jdk1.6.0_33\bin;C:\Program Files\QuickTime\QTSystem\;C:\Program Files\BSF4ooRexx;C:\Program Files\Java\Jre6\bin\client;C:\eclipse;
    USERNAME=ed83897
    OS=Windows_NT
    PROCESSOR_IDENTIFIER=x86 Family 6 Model 15 Stepping 13, GenuineIntel

    --------------- S Y S T E M ---------------

    OS: Windows XP Build 2600 Service Pack 3

    CPU:total 2 (2 cores per cpu, 1 threads per core) family 6 model 15 stepping 13, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3

    Memory: 4k page, physical 3405348k(1173576k free), swap 7416568k(4869832k free)

    vm_info: Java HotSpot(TM) Client VM (20.8-b03) for windows-x86 JRE (1.6.0_33-b05), built on Jun 28 2012 17:43:07 by "java_re" with MS VC++ 7.1 (VS2003)

    time: Tue Jan 08 11:24:31 2013
    elapsed time: 0 seconds

     
  • Erik Duijs

    Erik Duijs - 2013-01-08
    • status: pending-fixed --> open-fixed
     
  • Rony G. Flatscher

    Erik, are you sure that you have been using 20130107beta?

    I tried to recreate the crash on Windowxs XP with

    java version "1.6.0_30"
    Java(TM) SE Runtime Environment (build 1.6.0_30-b12)
    Java HotSpot(TM) Client VM (build 20.5-b03, mixed mode, sharing)

    However, I could not recreate it after running several times and even augmenting the THREADS field to 256!

    You could test from the command line:

    rexx -e "call bsf.cls;say .bsf4rexx~version"

    This should read 410.20130107.

    In addition you could run (from the bsf4oorexx directory):

    rexx infoBSF-oo.rxj

    which will give you further information and your environment settings, just to make sure that you are indeed using the new beta in your command line session.

     
  • Rony G. Flatscher

    Eric,

    sorry,

    rexx infoBSF-oo.rxj

    should read instead

    rexx install\infoBSF-oo.rxj

    And in case you have been using the latest beta, then please describe what you have been doing in order to get the crash, such that I can duplicate it locally.

    ---rony

     
  • Rony G. Flatscher

    Repeated, additional tests with THREADS=1024 and THREADS=2048 run stable.

     
  • Erik Duijs

    Erik Duijs - 2013-01-08

    Hi Rony,

    It's strange that I still see the issue while it seems fixed on your side.

    We don't really install BSF4ooRexx; we just include bsf-rexx-engine.jar in our classpath and BSF4ooRexx.dll in our library-path (BSF.CLS etc is referenced from the scripts themselves but is not used in the test program).

    BSF4ooRexx.dll on our library-path is dated 03-01-2013 18:11 (its the 32bit one)
    bsf-rexx-engine.jar on our class-path is dated 07-01-2013 17:49

    Doing "call bsf.cls;say .bsf4rexx~version" from within the test program gives me 410.20130107

    Rexx version:
    Open Object Rexx Version 4.1.1
    Build date: May 16 2012
    Addressing Mode: 32

    JRE version: 6.0_33-b05
    Java VM: Java HotSpot(TM) Client VM (20.8-b03 mixed mode, sharing windows-x86)

    Using this set-up, I did see that the issue related to returning wrong references (bug id 3581957) is solved using this beta, so all things considered I'm fairly confident that I'm using the latest version.

    I'm just running the test program (from within Eclipse) with 16 threads using the new .jar and .dll.
    It takes a few attempts to make it crash, and it usually keeps running fine when it started successfully, but every now and then starting the test program results in a JVM crash (about 1 out of 5 times on my machine).

    If there's something I can do to help, just let me know.

    Cheers,
    Erik

     
  • Rony G. Flatscher

    Yes, it is strange, unless an earlier version of BSF4ooRexx.dll gets effectively used on your system.

    What is the output of running:

    bsf4oorexx\install\infoBSF-oo.rxj

    If you run it, the second line should read:

    BSF4ooRexx (DLL/so): [410.20121230 org/rexxla/bsf/engines/rexx 32-bit]

    which is the BSF4ooRexx.dll that should be used as it contains the necessary fix.

    Should you be using the fixed version, then please start your great test program as:

    java -Xcheck:jni TestConcurrencyCrash

    In the case of a crash, please copy and paste any jni information which may be given by the JVM.

     
  • Erik Duijs

    Erik Duijs - 2013-01-08

    I think the version is correct, the complete output of infoBSF-oo.rxj is this:
    C:\eclipse\workspace\Bug_Reports\lib_beta20130107\bsf4oorexx>rexx install\infoBSF-oo.rxj
    Rexx interpreter: [REXX-ooRexx_4.1.1(MT) 6.03 16 May 2012]
    BSFManager [245.20090910]
    BSF4ooRexx (DLL/so): [410.20121230 org/rexxla/bsf/engines/rexx 32-bit]
    Java Rexx engine: [410.20120417 org.rexxla.bsf.engines.rexx (org.apache.bsf)]
    BSF.CLS version: [410.20130107]

    Java version: [1.6.0_37]
    Java bitness: [32]
    Java architecture: [x86]
    Java home dir: [C:\Program Files\Java\Jre6]

    Java library path: [C:\Program Files\ooRexx;C:\WINDOWS\Sun\Java\bin;C:\WINDOWS\system32;
    C:\WINDOWS;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\IBM\Pe
    rsonal Communications;C:\Program Files\IBM\Trace Facility;C:\Program Files\ooRexx;C:\WINDO
    WS\system32\WindowsPowerShell\v1.0;C:\Epoprogs\UserEXE;C:\Program Files\Gs\Gs9.02\Bin;C:\P
    rogram Files\Java\jdk1.6.0_33\bin;C:\Program Files\QuickTime\QTSystem\;C:\Program Files\Ja
    va\Jre6\bin\client;.]

    Java class path: [C:\Program Files\BSF4ooRexx\bsf-rexx-engine.jar;C:\Program Files\BSF
    4ooRexx\bsf-v400-20090910.jar;.]

    Java extension path: [C:\Program Files\Java\Jre6\lib\ext;C:\WINDOWS\Sun\Java\lib\ext]

    This Rexx script was invoked: [directly by Rexx which loaded Java]

    The following BSF-functions are registered with Rexx:
    [BSF]
    [BsfAttachToTID]
    [BsfCreateRexxProxy]
    [BsfDetach]
    [BsfDropFuncs]
    [BsfGetTID]
    [BsfInvokedBy]
    [BsfJavaException]
    [BsfLoadFuncs]
    [BsfLoadJava]
    [BsfQueryAllFunctions]
    [BsfQueryRegisteredFunctions]
    [BsfRawBytes]
    [BsfRexxProxy]
    [BsfShowErrorMessage]
    [BsfUnloadJava]
    [BsfVersion]

    When starting with -Xcheck:jni, it either works, or I get this:

    FATAL ERROR in native method: Wrong object class or methodID passed to JNI call
    at org.rexxla.bsf.engines.rexx.RexxAndJava.jniRexxCreateInterpreterInstance(Native Method)
    at org.rexxla.bsf.engines.rexx.RexxAndJava.createRexxInterpreterInstance(RexxAndJava.java:5585)
    at org.rexxla.bsf.engines.rexx.RexxEngine.eval(RexxEngine.java:250)
    at org.apache.bsf.util.BSFEngineImpl.exec(BSFEngineImpl.java:141)
    at org.rexxla.bsf.bugreport.TestConcurrencyCrash$1.run(TestConcurrencyCrash.java:23)
    at java.lang.Thread.run(Thread.java:662)

    Cheers,
    Erik

     
  • Rony G. Flatscher

    Hi Erik,

    thank you for your information!

    According to "infoBSF-oo.rxj" you are picking up an older RexxEngine implementaiton:

    Java Rexx engine: [410.20120417 org.rexxla.bsf.engines.rexx (org.apache.bsf)]

    where it should read:

    Java Rexx engine: [410.20130104 org.rexxla.bsf.engines.rexx (org.apache.bsf)]

    So something seems to be mixed up in your environment (maybe other files too).

    ---

    The other differences between your and my setup:

    - You are using a newer version of Java 1.6 (1.6.0_37) whereas I am using 1.6.0_30 (from the JDK). This should not make a dfiference, it is just a remark "for the record".

    - You are using ooRexx 4.1.1 whereas I am using ooRexx 4.1.2. Actually, this should not make a difference, it is just a remark "for the record".

    Cheers,

    ---rony

     
  • Erik Duijs

    Erik Duijs - 2013-01-08

    Hi Rony,

    I'm afraid you're right about my environment. BSF4ooRexx is now apparently installed system-wide here in the office (where it wasn't before), and unfortunately I was not aware of that. So my testing environment was picking up a .dll from an older version....
    I removed BSF4ooRexx from my system PATH and did a number of tests and all seems well now; no crashes anymore.

    Thank you very much for your time, and my apologies for the confusion.

    Erik

     
  • Rony G. Flatscher

    • status: open-fixed --> pending-fixed
     
  • Rony G. Flatscher

    Hi Erik,

    great (seriously!) that we could locate the cause in your environment!

    Please do not be afraid, things like this can happen from time to time.

    As your bug reports have been always accompanied by the greatest (seriously!) and shortest programs to demonstrate the reported bug, that also has allowed me in the past to find the causes of most of those bugs quite quickly, saving me a lot of research time to locate the spots where the bug came from.

    Again, glad that it has worked out now and that all tests pass on your side as well!

    Best regards,

    ---rony

     
  • Rony G. Flatscher

    • status: pending-fixed --> closed-fixed
     
  • Rony G. Flatscher

    • Group: Next Release --> version 411.20130714
     

Log in to post a comment.