#1161 Crash from object overlay usually during PackageClass::newRexx()

4.1.3
closed
Mark Miesfeld
None
none
1
2014-08-07
2013-02-12
No

When running multiple Rexx interpreter instances constantly in threads, eventually a crash occurs in RexxWaitForTermination(). This API is documented to have no effect in the new (4.0) C++ Rexx APIs.

A program, TestConcurrencyCrash4.java, accompanying a bug report in BSF4ooRexx (https://sourceforge.net/tracker/index.php?func=detail&aid=3604128&group_id=400586&atid=1660873) constantly creates Rexx interpreter instances in two different Java threads and executes a Rexx program from Java, using the BSF4ooRexx external function package.

Eventually a crash occurs that Java documents in a hs_err_pid???.log file, pointing at native code in rexx.dll, denoting the RexxWaitForTermination() function.

Environment:

ooRexx 4.1.2, 32-Bit
Java 1.6
latest BSF4ooRexx (from: https://sourceforge.net/projects/bsf4oorexx/files/GA/BSF4ooRexx-410.20130107-GA/)
Java-testprogram: TestConcurrencyCrash4.java from the above bug report

After having BSF4ooRexx installed, and compiling of the Java-Program, one may run it using "java TestConcurrencyCrash4" from the command line to get at the crash.

1 Attachments

Related

Bugs: #1161

Discussion

1 2 > >> (Page 1 of 2)
  • Mark Miesfeld
    Mark Miesfeld
    2013-06-09

    • summary: Crash in RexxWaitForTermination() --> Crash from object overlay usually during PackageClass::newRexx()
     
  • Mark Miesfeld
    Mark Miesfeld
    2013-06-09

    This crash has nothing to do with RexxWaitForTermination.

    I spent quite some time looking at this and I see that at the point of the crash, an object has had some fields overlaid with bad values.

    For me 90% of the time this happens in PackageClass::newRexx() with the object being the RexxString resolvedName

    RexxString *resolvedName = instance->resolveProgramName(nameString, OREF_NULL, OREF_NULL);

    In newRexx() the sequence goes loadRequires -> addRequires -> RexxDirectory::put() with the index being the resolvedName object.

    From there it goes stringArgument() -> RexxObject::requiredString() -> RexxInternalObject::isBaseClass() -> RexxBehaviour::isPrimitive()

    It crashes there with the this pointer in isPrimitive() being null. If you go back up the stack a bit, the behaviour field of the RexxInternalObject of the resolvedName object set is NULL.

    When I look, the _vfptr always looks good, and usually some or all of the header fields are incorrect. Although in this example they are correct.

    • RexxInternalObject {header={...} behaviour=0x0000000000000000 } RexxInternalObject
    • RexxVirtualBase {...} RexxVirtualBase
    • __vfptr 0x000007feef177d78 const RexxString::`vftable' *
    • header {objectSize=0x00000000000000a0 flags=0x0022 sizePadding=0x0000000000000022 } ObjectHeader
      objectSize 0x00000000000000a0 unsigned int64
      flags 0x0022 unsigned short
      sizePadding 0x0000000000000022 unsigned
      int64
    • behaviour 0x0000000000000000 {classType=??? behaviourFlags=??? scopes=??? ...} RexxBehaviour *

    For instance, header might be:

    • header {objectSize=0x00000000000000a0 flags=0x0021 sizePadding=0x000007fffd390021 } ObjectHeader
      objectSize 0x00000000000000a0 unsigned int64
      flags 0x0021 unsigned short
      sizePadding 0x000007fffd390021 unsigned
      int64

    Some times the crash is not in this path, with it usually being in markObjectsMain(). It crashes at this line:

        markObject->live(liveMark);
    

    Looking at markObject in the debugger, I see many of the fields of the object with bad values.

    I haven't been able to make any progress in determining when or how the resolvedName object gets overlaid.

     
  • Rick McGuire
    Rick McGuire
    2013-06-09

    If this is consistently the same object, then it is likely this object
    needs to be protected from garbage collection after it is created.

    On Sun, Jun 9, 2013 at 2:38 PM, Mark Miesfeld miesfeld@users.sf.net wrote:

    This crash has nothing to do with RexxWaitForTermination.

    I spent quite some time looking at this and I see that at the point of the
    crash, an object has had some fields overlaid with bad values.

    For me 90% of the time this happens in PackageClass::newRexx() with the
    object being the RexxString resolvedName

    RexxString *resolvedName = instance->resolveProgramName(nameString,
    OREF_NULL, OREF_NULL);

    In newRexx() the sequence goes loadRequires -> addRequires ->
    RexxDirectory::put() with the index being the resolvedName object.

    From there it goes stringArgument() -> RexxObject::requiredString() ->
    RexxInternalObject::isBaseClass() -> RexxBehaviour::isPrimitive()

    It crashes there with the this pointer in isPrimitive() being null. If
    you go back up the stack a bit, the behaviour field of the
    RexxInternalObject of the resolvedName object set is NULL.

    When I look, the _vfptr always looks good, and usually some or all of the
    header fields are incorrect. Although in this example they are correct.

    • RexxInternalObject {header={...}
      behaviour=0x0000000000000000 } RexxInternalObject
    • RexxVirtualBase {...} RexxVirtualBase
    • __vfptr 0x000007feef177d78 const RexxString::`vftable' *
    • header {objectSize=0x00000000000000a0 flags=0x0022
      sizePadding=0x0000000000000022 } ObjectHeader
      objectSize 0x00000000000000a0 unsigned int64
      flags 0x0022 unsigned short
      sizePadding 0x0000000000000022 unsigned
      int64
    • behaviour 0x0000000000000000 {classType=???
      behaviourFlags=??? scopes=??? ...} RexxBehaviour *

    For instance, header might be:

    • header {objectSize=0x00000000000000a0 flags=0x0021
      sizePadding=0x000007fffd390021 } ObjectHeader
      objectSize 0x00000000000000a0 unsigned int64
      flags 0x0021 unsigned short
      sizePadding 0x000007fffd390021 unsigned
      int64

    Some times the crash is not in this path, with it usually being in
    markObjectsMain(). It crashes at this line:

        markObject->live(liveMark);
    

    Looking at markObject in the debugger, I see many of the fields of the
    object with bad values.

    I haven't been able to make any progress in determining when or how the
    resolvedName object gets overlaid.


    [bugs:#1161] Crash from object overlay usually during
    PackageClass::newRexx()

    Status: open
    Created: Tue Feb 12, 2013 03:37 PM UTC by Rony G. Flatscher
    Last Updated: Thu May 30, 2013 04:26 PM UTC
    Owner: nobody

    When running multiple Rexx interpreter instances constantly in threads,
    eventually a crash occurs in RexxWaitForTermination(). This API is
    documented to have no effect in the new (4.0) C++ Rexx APIs.

    A program, TestConcurrencyCrash4.java, accompanying a bug report in
    BSF4ooRexx (
    https://sourceforge.net/tracker/index.php?func=detail&aid=3604128&group_id=400586&atid=1660873)
    constantly creates Rexx interpreter instances in two different Java threads
    and executes a Rexx program from Java, using the BSF4ooRexx external
    function package.

    Eventually a crash occurs that Java documents in a hs_err_pid???.log file,
    pointing at native code in rexx.dll, denoting the RexxWaitForTermination()
    function.

    Environment:

    ooRexx 4.1.2, 32-Bit
    Java 1.6
    latest BSF4ooRexx (from:
    

    https://sourceforge.net/projects/bsf4oorexx/files/GA/BSF4ooRexx-410.20130107-GA/
    )
    Java-testprogram: TestConcurrencyCrash4.java from the above bug report

    After having BSF4ooRexx installed, and compiling of the Java-Program, one
    may run it using "java TestConcurrencyCrash4" from the command line to get
    at the crash.


    Sent from sourceforge.net because you indicated interest in <
    https://sourceforge.net/p/oorexx/bugs/1161/>

    To unsubscribe from further messages, please visit <
    https://sourceforge.net/auth/subscriptions/>

     

    Related

    Bugs: #1161

  • Mark Miesfeld
    Mark Miesfeld
    2013-06-09

    • status: open --> pending
    • assigned_to: Mark Miesfeld
    • Group: 4.1.2 --> vX.X.X
     
  • Mark Miesfeld
    Mark Miesfeld
    2013-06-09

    Committed revision 9301. [r9301] trunk

    I believe this commit fixes the specific crash initially reported here and the crash I could reproduce. Prior to the code change I could reproduce the crash 100% of the times I ran the test program, within a reasonably short period of time.

    After the code change, I ran the test around 30 times and never saw that crash.

    However, if the program is not stopped after some amount of time it eventually consumes all memory on my system. At which point the program needs to be halted. Out of the 30 tests, twice the test program did crash after the operating system warned that the system was unstable and could not continue. Twice the Visual Studio debugger itself crashed, and once Chrome crashed.

    I think it is unreasonable to expect any program to be stable after the point the operating system warns that memory is so low the system can not continue. Therefore I don't give any significance to the 2 crashes I did see.

     

    Related

    Commit: [r9301]

  • Thank you very much, indeed!


    Ad constantly consuming memory: I have tested over the weekend a comparable program written in pure C++, creating constantly Rexx interpreter instances to run the same Rexx program and doing a Terminate() thereafter. This program runs forever and stable (had to even limit the loop to 750,000 Rexx interpreter creations :) ) !

    So over the weekend I have been trying to look into the BSF4ooRexx framework to find a reason why the memory consumption constantly is raised there, running the same Rexx program with the same logic. I think I found the culprit in this case being in BSF4ooRexx.cc (it seems that it is not really terminating a Rexx interpreter instance in the native code under certain conditions), which I further need to debug.

    So to make a long story short: I think the problem with running out of system resources is not linked to ooRexx, but to BSF4ooRexx. Will update the tracker items once I know for sure.

     
1 2 > >> (Page 1 of 2)


Anonymous


Cancel   Add attachments