#13 Potential crash when starting concurrently

411.20130714
closed-fixed
C++ (2)
5
2013-10-08
2012-11-29
Erik Duijs
No

When concurrently starting a REXX script from java, BSF4ooRexx seems prone to crashing (taking the JVM down with it).

See the attached test program to reproduce and a crash report. It doesn't consistently crash 100% of the time, but usually it does. Raising the THREADS constant seems to increase the likelyhood of the crash.
The vulnerability seems to be mainly while starting the REXX script: If it does start successfully, it seems to keep running as it should.

Discussion

1 2 > >> (Page 1 of 2)
  • Erik Duijs
    Erik Duijs
    2012-11-29

    Test program to reproduce

     
  • Erik Duijs
    Erik Duijs
    2012-11-29

    JVM crash report

     
    Attachments
  • Thanks, Erik, will look into it sometimes in the next week. So anyone beating me, is highly welcome! ;)

     
    • labels: --> C++
    • assigned_to: nobody --> orexx
    • milestone: --> Next Release
    • status: open --> pending-fixed
     
  • Erik Duijs
    Erik Duijs
    2013-01-08

    I've tested it with the 20130107 beta version, and unfortunately I could still crash the JVM with the TestConcurrencyCrash.java test program (although perhaps it happens less frequently).

    This is in the JVM crash log:
    #
    # A fatal error has been detected by the Java Runtime Environment:
    #
    # EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x6d95744a, pid=4620, tid=1672
    #
    # JRE version: 6.0_33-b05
    # Java VM: Java HotSpot(TM) Client VM (20.8-b03 mixed mode, sharing windows-x86 )
    # Problematic frame:
    # V [jvm.dll+0x9744a]
    #
    # If you would like to submit a bug report, please visit:
    # http://java.sun.com/webapps/bugreport/crash.jsp
    #

    --------------- T H R E A D ---------------

    Current thread (0x02b8f000): JavaThread "Thread-1" [_thread_in_vm, id=1672, stack(0x02ed0000,0x02f20000)]

    siginfo: ExceptionCode=0xc0000005, reading address 0x00000000

    Registers:
    EAX=0x00000000, EBX=0x02b8f128, ECX=0x00000006, EDX=0x02ba2c68
    ESP=0x02f1f484, EBP=0x02f1f4cc, ESI=0x02b8f000, EDI=0x00000000
    EIP=0x6d95744a, EFLAGS=0x00010246

    Top of Stack: (sp=0x02f1f484)
    0x02f1f484: 00000000 02b8f128 000a6b80 02f1f9c0
    0x02f1f494: 02b8f128 000a6b80 000a6000 000a6000
    0x02f1f4a4: 02f1f490 ffffffff 02f1f4ec 7c839ab0
    0x02f1f4b4: 7c809f08 ffffffff 02b8f000 00000000
    0x02f1f4c4: 000a6b80 7c810068 00000048 100012de
    0x02f1f4d4: 02b8f128 02ba2c68 00000000 02f1f4f4
    0x02f1f4e4: 10002788 02b8f128 02ba2c68 00000000
    0x02f1f4f4: 02f1f9c0 02b8f128 00000048 000a6b80

    Instructions: (pc=0x6d95744a)
    0x6d95742a: ff 8b 46 04 83 c4 08 85 c0 89 75 f0 c7 45 f4 00
    0x6d95743a: 00 00 00 74 08 8d 4d f0 e8 49 a2 09 00 8b 7d 10
    0x6d95744a: 8b 07 c7 45 e0 0c 00 00 00 8b 48 08 0f b7 51 2a
    0x6d95745a: 8b 40 0c 8b 4c 90 28 51 56 8d 4d c4 e8 05 8c 07

    Register to memory mapping:

    EAX=0x00000000 is an unknown value
    EBX=0x02b8f128 is an unknown value
    ECX=0x00000006 is an unknown value
    EDX=0x02ba2c68 is an unknown value
    ESP=0x02f1f484 is pointing into the stack for thread: 0x02b8f000
    EBP=0x02f1f4cc is pointing into the stack for thread: 0x02b8f000
    ESI=0x02b8f000 is a thread
    EDI=0x00000000 is an unknown value

    Stack: [0x02ed0000,0x02f20000], sp=0x02f1f484, free space=317k
    Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
    V [jvm.dll+0x9744a]

    Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
    j org.rexxla.bsf.engines.rexx.RexxAndJava.jniRexxCreateInterpreterInstance([Ljava/lang/Object;)Ljava/lang/String;+0
    j org.rexxla.bsf.engines.rexx.RexxAndJava.createRexxInterpreterInstance(Lorg/rexxla/bsf/engines/rexx/RexxConfiguration;)Ljava/lang/String;+65
    j org.rexxla.bsf.engines.rexx.RexxEngine.eval(Ljava/lang/String;IILjava/lang/Object;)Ljava/lang/Object;+111
    j org.apache.bsf.util.BSFEngineImpl.exec(Ljava/lang/String;IILjava/lang/Object;)V+6
    j org.rexxla.bsf.bugreport.TestConcurrencyCrash$1.run()V+33
    j java.lang.Thread.run()V+11
    v ~StubRoutines::call_stub

    --------------- P R O C E S S ---------------

    Java Threads: ( => current thread )
    0x003c6c00 JavaThread "DestroyJavaVM" [_thread_blocked, id=5824, stack(0x008c0000,0x00910000)]
    0x02bc5000 JavaThread "Thread-15" [_thread_in_native, id=1740, stack(0x03330000,0x03380000)]
    0x02bc3000 JavaThread "Thread-14" [_thread_in_native, id=3836, stack(0x032e0000,0x03330000)]
    0x02bc1400 JavaThread "Thread-13" [_thread_in_native, id=5196, stack(0x03290000,0x032e0000)]
    0x02bbd000 JavaThread "Thread-12" [_thread_in_native, id=5404, stack(0x03240000,0x03290000)]
    0x02bae000 JavaThread "Thread-11" [_thread_in_native, id=5796, stack(0x031f0000,0x03240000)]
    0x02bac800 JavaThread "Thread-10" [_thread_in_native, id=4008, stack(0x031a0000,0x031f0000)]
    0x02bab000 JavaThread "Thread-9" [_thread_in_native, id=3840, stack(0x03150000,0x031a0000)]
    0x02baa000 JavaThread "Thread-8" [_thread_in_native, id=4056, stack(0x03100000,0x03150000)]
    0x02ba5000 JavaThread "Thread-7" [_thread_in_native, id=4176, stack(0x030b0000,0x03100000)]
    0x02ba3400 JavaThread "Thread-6" [_thread_in_native, id=476, stack(0x03060000,0x030b0000)]
    0x02ba1800 JavaThread "Thread-5" [_thread_in_native, id=1216, stack(0x03010000,0x03060000)]
    0x02ba0400 JavaThread "Thread-4" [_thread_in_native, id=5992, stack(0x02fc0000,0x03010000)]
    0x02b9ec00 JavaThread "Thread-3" [_thread_in_native, id=5988, stack(0x02f70000,0x02fc0000)]
    0x02b90400 JavaThread "Thread-2" [_thread_in_native, id=4540, stack(0x02f20000,0x02f70000)]
    =>0x02b8f000 JavaThread "Thread-1" [_thread_in_vm, id=1672, stack(0x02ed0000,0x02f20000)]
    0x02b9cc00 JavaThread "Thread-0" [_thread_in_native, id=948, stack(0x02e80000,0x02ed0000)]
    0x02b71c00 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=1172, stack(0x02de0000,0x02e30000)]
    0x02b63000 JavaThread "C1 CompilerThread0" daemon [_thread_blocked, id=1132, stack(0x02d90000,0x02de0000)]
    0x02b61400 JavaThread "Attach Listener" daemon [_thread_blocked, id=3084, stack(0x02d40000,0x02d90000)]
    0x02b60000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=5372, stack(0x02cf0000,0x02d40000)]
    0x02b5b800 JavaThread "Finalizer" daemon [_thread_blocked, id=2536, stack(0x02ca0000,0x02cf0000)]
    0x02b56c00 JavaThread "Reference Handler" daemon [_thread_blocked, id=5800, stack(0x02c50000,0x02ca0000)]

    Other Threads:
    0x02b1ac00 VMThread [stack: 0x02c00000,0x02c50000] [id=5952]
    0x02b7d400 WatcherThread [stack: 0x02e30000,0x02e80000] [id=6036]

    VM state:not at safepoint (normal execution)

    VM Mutex/Monitor currently owned by a thread: None

    Heap
    def new generation total 4928K, used 2564K [0x22990000, 0x22ee0000, 0x27ee0000)
    eden space 4416K, 58% used [0x22990000, 0x22c11180, 0x22de0000)
    from space 512K, 0% used [0x22de0000, 0x22de0000, 0x22e60000)
    to space 512K, 0% used [0x22e60000, 0x22e60000, 0x22ee0000)
    tenured generation total 10944K, used 0K [0x27ee0000, 0x28990000, 0x32990000)
    the space 10944K, 0% used [0x27ee0000, 0x27ee0000, 0x27ee0200, 0x28990000)
    compacting perm gen total 12288K, used 272K [0x32990000, 0x33590000, 0x36990000)
    the space 12288K, 2% used [0x32990000, 0x329d4140, 0x329d4200, 0x33590000)
    ro space 10240K, 54% used [0x36990000, 0x36f0f260, 0x36f0f400, 0x37390000)
    rw space 12288K, 55% used [0x37390000, 0x37a33000, 0x37a33000, 0x37f90000)

    Code Cache [0x00920000, 0x009a8000, 0x02920000)
    total_blobs=181 nmethods=38 adapters=79 free_code_cache=33020032 largest_free_block=0

    Dynamic libraries:
    0x00400000 - 0x00425000 C:\Program Files\Java\jdk1.6.0_33\jre\bin\javaw.exe
    0x7c900000 - 0x7c9b2000 C:\WINDOWS\system32\ntdll.dll
    0x7c800000 - 0x7c8f6000 C:\WINDOWS\system32\kernel32.dll
    0x77dd0000 - 0x77e6b000 C:\WINDOWS\system32\ADVAPI32.dll
    0x77e70000 - 0x77f03000 C:\WINDOWS\system32\RPCRT4.dll
    0x77fe0000 - 0x77ff1000 C:\WINDOWS\system32\Secur32.dll
    0x7e410000 - 0x7e4a1000 C:\WINDOWS\system32\USER32.dll
    0x77f10000 - 0x77f59000 C:\WINDOWS\system32\GDI32.dll
    0x76390000 - 0x763ad000 C:\WINDOWS\system32\IMM32.DLL
    0x629c0000 - 0x629c9000 C:\WINDOWS\system32\LPK.DLL
    0x74d90000 - 0x74dfb000 C:\WINDOWS\system32\USP10.dll
    0x641f0000 - 0x641fc000 C:\PROGRA~1\NetInst\NiAMH.dll
    0x7c340000 - 0x7c396000 C:\Program Files\Java\jdk1.6.0_33\jre\bin\msvcr71.dll
    0x6d8c0000 - 0x6db6f000 C:\Program Files\Java\jdk1.6.0_33\jre\bin\client\jvm.dll
    0x76b40000 - 0x76b6d000 C:\WINDOWS\system32\WINMM.dll
    0x6d870000 - 0x6d87c000 C:\Program Files\Java\jdk1.6.0_33\jre\bin\verify.dll
    0x6d3d0000 - 0x6d3ef000 C:\Program Files\Java\jdk1.6.0_33\jre\bin\java.dll
    0x6d8b0000 - 0x6d8bf000 C:\Program Files\Java\jdk1.6.0_33\jre\bin\zip.dll
    0x10000000 - 0x10022000 C:\Program Files\BSF4ooRexx\BSF4ooRexx.dll
    0x03580000 - 0x03652000 C:\Program Files\ooRexx\rexx.dll
    0x773d0000 - 0x774d3000 C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.2600.6028_x-ww_61e65202\COMCTL32.dll
    0x77c10000 - 0x77c68000 C:\WINDOWS\system32\msvcrt.dll
    0x77f60000 - 0x77fd6000 C:\WINDOWS\system32\SHLWAPI.dll
    0x03660000 - 0x03685000 C:\Program Files\ooRexx\REXXAPI.dll
    0x71ad0000 - 0x71ad9000 C:\WINDOWS\system32\WSOCK32.dll
    0x71ab0000 - 0x71ac7000 C:\WINDOWS\system32\WS2_32.dll
    0x71aa0000 - 0x71aa8000 C:\WINDOWS\system32\WS2HELP.dll
    0x6d0b0000 - 0x6d1fc000 C:\Program Files\Java\jdk1.6.0_33\jre\bin\awt.dll
    0x73000000 - 0x73026000 C:\WINDOWS\system32\WINSPOOL.DRV
    0x774e0000 - 0x7761e000 C:\WINDOWS\system32\ole32.dll
    0x76bf0000 - 0x76bfb000 C:\WINDOWS\system32\PSAPI.DLL
    0x6b1f0000 - 0x6b214000 C:\Program Files\ooRexx\OREXXOLE.dll
    0x77120000 - 0x771ab000 C:\WINDOWS\system32\OLEAUT32.dll
    0x6b120000 - 0x6b155000 C:\Program Files\ooRexx\rexxutil.dll
    0x71a50000 - 0x71a8f000 C:\WINDOWS\system32\mswsock.dll
    0x662b0000 - 0x66308000 C:\WINDOWS\system32\hnetcfg.dll
    0x71a90000 - 0x71a98000 C:\WINDOWS\System32\wshtcpip.dll
    0x76f20000 - 0x76f47000 C:\WINDOWS\system32\DNSAPI.dll

    VM Arguments:
    jvm_args: -Dfile.encoding=Cp1252
    java_command: org.rexxla.bsf.bugreport.TestConcurrencyCrash
    Launcher Type: SUN_STANDARD

    Environment Variables:
    CLASSPATH=C:\Program Files\BSF4ooRexx\bsf-rexx-engine.jar;C:\Program Files\BSF4ooRexx\bsf-v400-20090910.jar;.
    PATH=C:/Program Files/Java/Jre6/bin/client;C:/Program Files/Java/Jre6/bin;C:/Program Files/Java/Jre6/lib/i386;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\IBM\Personal Communications;C:\Program Files\IBM\Trace Facility;C:\Program Files\ooRexx;C:\WINDOWS\system32\WindowsPowerShell\v1.0;C:\Epoprogs\UserEXE;C:\Program Files\Gs\Gs9.02\Bin;C:\Program Files\Java\jdk1.6.0_33\bin;C:\Program Files\QuickTime\QTSystem\;C:\Program Files\BSF4ooRexx;C:\Program Files\Java\Jre6\bin\client;C:\eclipse;
    USERNAME=ed83897
    OS=Windows_NT
    PROCESSOR_IDENTIFIER=x86 Family 6 Model 15 Stepping 13, GenuineIntel

    --------------- S Y S T E M ---------------

    OS: Windows XP Build 2600 Service Pack 3

    CPU:total 2 (2 cores per cpu, 1 threads per core) family 6 model 15 stepping 13, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3

    Memory: 4k page, physical 3405348k(1173576k free), swap 7416568k(4869832k free)

    vm_info: Java HotSpot(TM) Client VM (20.8-b03) for windows-x86 JRE (1.6.0_33-b05), built on Jun 28 2012 17:43:07 by "java_re" with MS VC++ 7.1 (VS2003)

    time: Tue Jan 08 11:24:31 2013
    elapsed time: 0 seconds

     
  • Erik Duijs
    Erik Duijs
    2013-01-08

    • status: pending-fixed --> open-fixed
     
  • Erik, are you sure that you have been using 20130107beta?

    I tried to recreate the crash on Windowxs XP with

    java version "1.6.0_30"
    Java(TM) SE Runtime Environment (build 1.6.0_30-b12)
    Java HotSpot(TM) Client VM (build 20.5-b03, mixed mode, sharing)

    However, I could not recreate it after running several times and even augmenting the THREADS field to 256!

    You could test from the command line:

    rexx -e "call bsf.cls;say .bsf4rexx~version"

    This should read 410.20130107.

    In addition you could run (from the bsf4oorexx directory):

    rexx infoBSF-oo.rxj

    which will give you further information and your environment settings, just to make sure that you are indeed using the new beta in your command line session.

     
  • Eric,

    sorry,

    rexx infoBSF-oo.rxj

    should read instead

    rexx install\infoBSF-oo.rxj

    And in case you have been using the latest beta, then please describe what you have been doing in order to get the crash, such that I can duplicate it locally.

    ---rony

     
  • Repeated, additional tests with THREADS=1024 and THREADS=2048 run stable.

     
  • Erik Duijs
    Erik Duijs
    2013-01-08

    Hi Rony,

    It's strange that I still see the issue while it seems fixed on your side.

    We don't really install BSF4ooRexx; we just include bsf-rexx-engine.jar in our classpath and BSF4ooRexx.dll in our library-path (BSF.CLS etc is referenced from the scripts themselves but is not used in the test program).

    BSF4ooRexx.dll on our library-path is dated 03-01-2013 18:11 (its the 32bit one)
    bsf-rexx-engine.jar on our class-path is dated 07-01-2013 17:49

    Doing "call bsf.cls;say .bsf4rexx~version" from within the test program gives me 410.20130107

    Rexx version:
    Open Object Rexx Version 4.1.1
    Build date: May 16 2012
    Addressing Mode: 32

    JRE version: 6.0_33-b05
    Java VM: Java HotSpot(TM) Client VM (20.8-b03 mixed mode, sharing windows-x86)

    Using this set-up, I did see that the issue related to returning wrong references (bug id 3581957) is solved using this beta, so all things considered I'm fairly confident that I'm using the latest version.

    I'm just running the test program (from within Eclipse) with 16 threads using the new .jar and .dll.
    It takes a few attempts to make it crash, and it usually keeps running fine when it started successfully, but every now and then starting the test program results in a JVM crash (about 1 out of 5 times on my machine).

    If there's something I can do to help, just let me know.

    Cheers,
    Erik

     
1 2 > >> (Page 1 of 2)