#2078 Interpreter::resourceLock is never locked; static initialization order clears its created flag

[Moritz Hoffmann]

On Unix, Interpreter::resourceLock is never in a usable state, for the entire
life of every process. Every ResourceSection silently guards nothing.

Mechanism

_rexx_init() in interpreter/platform/unix/SystemInitialization.cpp is an ELF
constructor. It calls Interpreter::createLocks(), which calls
resourceLock.create(true), setting created = true.

SysMutex Interpreter::resourceLock; is a namespace-scope object in
Interpreter.cpp. Its dynamic initializer, SysMutex() : created(false), runs
in link order relative to that constructor. In the builds I tested it runs
after, and resets the flag.

SysMutex::request() returns false when !created, so from that point on
every acquire fails and every critical section is unprotected.

Evidence

An ordering probe printing this in the constructor and in createLocks():

*** createLocks() created resourceLock at 0x...9e0
*** SysMutex() ctor at 0x...9e0             <- runs afterwards, resets created

A probe distinguishing the two failure branches of request() reported
NOT-CREATED for 60 of 60 acquire failures in one test group, at these
call sites:

InterpreterInstance::poolActivity
InterpreterInstance::terminate  (via removeInactiveActivities)
InterpreterInstance::spawnActivity
ActivityManager::createNewActivity

Interpreter::dispatchLock was declared in the same file and was dead in the
same way. A probe on the acquire result counts 4855 DispatchSection
constructions that failed to lock in a single GUARD test group run, so the
activity dispatch queue has no protection at all.

Consequences

A deterministic crash. Interpreter::startInterpreter() guards one-time
bootstrap with:

ResourceSection lock;
if (!isActive())
{
    ... memoryObject.initialize(), restore the image, build the local server ...
}

With the lock inert, that test is unsynchronised. Several threads calling
RexxCreateInterpreter concurrently all pass it and all restore the image into
the same global memoryObject. Eight threads creating and terminating
instances segfaults 15 runs out of 15; gdb shows every thread inside
startInterpreter -> MemoryObject::initialize -> restoreImage simultaneously. A
counter in the bootstrap body records 8 entries where there must be 1.

A test binary for this is attached as provoke_locks.cpp. The rexx command
line tool creates one instance on one thread, which is why this has not been
seen in normal use; it needs the embedding C API driven from several threads.

Data races. ThreadSanitizer reports InterpreterInstance::allActivities
and ActivityManager::allActivities raced between poolActivity,
removeInactiveActivities and activityEnded, all of which take
ResourceSection in the source. Across six targeted test groups, making the
lock work takes data races from 45 to 19.

This also explains the unresolved half of bug #1734. After the r12435 fix,
erich_st reported "instead of the intermittent hangs there are intermittent
crashes", with ArrayClass::findSingleIndexItem:2086 <- removeItem:2221 <- poolActivity. That is the same race, and it was not reproducible for bigrixx
because whether the lock works at all depends on link order.

Suggested fix

Make the lock a function-local static, initialized on first use, so the
ordering cannot invert:

SysMutex &Interpreter::resourceLock()
{
    static SysMutex theLock;
    return theLock;
}

ActivityManager::kernelSemaphore, terminationSem and the dispatch lock are
namespace-scope objects with the same exposure, safe today only because of link
order, and want the same treatment.

One thing to be aware of before applying it

With the resource lock actually held, ThreadSanitizer reports a lock order
inversion it could not see before: createInterpreterInstance holds the
resource lock across startInterpreter(), which recurses into
createInterpreterInstance -> getRootActivity() -> lockKernel(), requesting the
kernel lock while holding the resource lock. Interpreter.hpp forbids exactly
that, and the comment in createInterpreterInstance requires the opposite. The
two rules contradict each other and only coexisted because the lock was inert.

This is the same lock pair as the #1734 hang, and startInterpreter still has
InstanceBlock inside a ResourceSection — the pattern r12435 removed from
terminateInterpreter. It is not reachable as a deadlock today, because
active is never reset and so the bootstrap body runs once per process, before
any other thread can hold the kernel lock. Measured: no hang at 2, 4, 8 or 16
threads. It becomes reachable if startup is ever made re-entrant.

Full test suite with the fix: 24372 tests, 387294 assertions, no new failures.

[Per Olov Jonsson]

Dear Moritz,

I think the agreed procedure is to change from open to pendling as long as we are on beta. With a new release the pending items are summed up in a document and all pending items changed to closed.

Jenkins can now run all tests on FreeBSD without any crashes or errors so your work seems to have fixed a LOT of problems on the BSDs and that is great!

One funny/strange thing is that we now have the same problem on FreeBSD as on Windows in a VM; the test suite does not finish completely and times out eventually. There seems to be a process that is left dangling after the test completion.

Von meinem iPhone gesendet
P.O. Jonsson

Am 03.08.2026 um 23:43 schrieb Moritz Hoffmann antiguru@users.sourceforge.net:

- status: open --> closed
- assigned_to: Moritz Hoffmann
- Comment:

Fixed in r13185 + r13186.

---

[bugs:#2078] Interpreter::resourceLock is never locked; static initialization order clears its created flag

Status: closed
Group: 5.3.0
Created: Sun Jul 26, 2026 09:41 AM UTC by Moritz Hoffmann
Last Updated: Sun Jul 26, 2026 09:42 AM UTC
Owner: Moritz Hoffmann
Attachments:

• provoke_locks.cpp (sourceforge.net) (9.4 kB; application/octet-stream)
• provoke_locks.rex (sourceforge.net) (524 Bytes; application/octet-stream)

On Unix, Interpreter::resourceLock is never in a usable state, for the entire
life of every process. Every ResourceSection silently guards nothing.

Mechanism

_rexx_init() in interpreter/platform/unix/SystemInitialization.cpp is an ELF
constructor. It calls Interpreter::createLocks(), which calls
resourceLock.create(true), setting created = true.

SysMutex Interpreter::resourceLock; is a namespace-scope object in
Interpreter.cpp. Its dynamic initializer, SysMutex() : created(false), runs
in link order relative to that constructor. In the builds I tested it runs
after, and resets the flag.

SysMutex::request() returns false when !created, so from that point on
every acquire fails and every critical section is unprotected.

Evidence

An ordering probe printing this in the constructor and in createLocks():

~~~
*** createLocks() created resourceLock at 0x...9e0
*** SysMutex() ctor at 0x...9e0 <- runs afterwards, resets created
~~~

A probe distinguishing the two failure branches of request() reported
NOT-CREATED for 60 of 60 acquire failures in one test group, at these
call sites:

~~~
InterpreterInstance::poolActivity
InterpreterInstance::terminate (via removeInactiveActivities)
InterpreterInstance::spawnActivity
ActivityManager::createNewActivity
~~~

Interpreter::dispatchLock was declared in the same file and was dead in the
same way. A probe on the acquire result counts 4855 DispatchSection
constructions that failed to lock in a single GUARD test group run, so the
activity dispatch queue has no protection at all.

Consequences

A deterministic crash. Interpreter::startInterpreter() guards one-time
bootstrap with:

~~~cpp
ResourceSection lock;
if (!isActive())
{
... memoryObject.initialize(), restore the image, build the local server ...
}
~~~

With the lock inert, that test is unsynchronised. Several threads calling
RexxCreateInterpreter concurrently all pass it and all restore the image into
the same global memoryObject. Eight threads creating and terminating
instances segfaults 15 runs out of 15; gdb shows every thread inside
startInterpreter -> MemoryObject::initialize -> restoreImage simultaneously. A
counter in the bootstrap body records 8 entries where there must be 1.

A test binary for this is attached as provoke_locks.cpp. The rexx command
line tool creates one instance on one thread, which is why this has not been
seen in normal use; it needs the embedding C API driven from several threads.

Data races. ThreadSanitizer reports InterpreterInstance::allActivities
and ActivityManager::allActivities raced between poolActivity,
removeInactiveActivities and activityEnded, all of which take
ResourceSection in the source. Across six targeted test groups, making the
lock work takes data races from 45 to 19.

This also explains the unresolved half of bug #1734. After the r12435 fix,
erich_st reported "instead of the intermittent hangs there are intermittent
crashes", with ArrayClass::findSingleIndexItem:2086 <- removeItem:2221 <- poolActivity. That is the same race, and it was not reproducible for bigrixx
because whether the lock works at all depends on link order.

Suggested fix

Make the lock a function-local static, initialized on first use, so the
ordering cannot invert:

~~~cpp
SysMutex &Interpreter::resourceLock()
{
static SysMutex theLock;
return theLock;
}
~~~

ActivityManager::kernelSemaphore, terminationSem and the dispatch lock are
namespace-scope objects with the same exposure, safe today only because of link
order, and want the same treatment.

One thing to be aware of before applying it

With the resource lock actually held, ThreadSanitizer reports a lock order
inversion it could not see before: createInterpreterInstance holds the
resource lock across startInterpreter(), which recurses into
createInterpreterInstance -> getRootActivity() -> lockKernel(), requesting the
kernel lock while holding the resource lock. Interpreter.hpp forbids exactly
that, and the comment in createInterpreterInstance requires the opposite. The
two rules contradict each other and only coexisted because the lock was inert.

This is the same lock pair as the #1734 hang, and startInterpreter still has
InstanceBlock inside a ResourceSection — the pattern r12435 removed from
terminateInterpreter. It is not reachable as a deadlock today, because
active is never reset and so the bootstrap body runs once per process, before
any other thread can hold the kernel lock. Measured: no hang at 2, 4, 8 or 16
threads. It becomes reachable if startup is ever made re-entrant.

Full test suite with the fix: 24372 tests, 387294 assertions, no new failures.

---

Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/oorexx/bugs/2078/

To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/

[m-stgt]

I think the agreed procedure is to change from open to pendling as long as we are on beta. With a new release the pending items are summed up in a document and all pending items changed to closed.

Since when this is a agreed procedure?
Why I ask? CHANGES.txt what comes with ooRexx 5.2.0 repeated items already listed in CHANGES.txt of V5.1.0 -- for details see https://sourceforge.net/p/oorexx/discussion/408478/thread/e33e7a9161/

[Rick McGuire]

It has been that way for a long time. The Pending status is how we identify
what changes need to be listed in CHANGES.txt when a new release is
prepared.

Rick

On Tue, Aug 4, 2026 at 1:01 PM m-stgt m-stgt@users.sourceforge.net wrote:

I think the agreed procedure is to change from open to pendling as long
as we are on beta. With a new release the pending items are summed up in a
document and all pending items changed to closed.

Since when this is a agreed procedure?
Why I ask? CHANGES.txt what comes with ooRexx 5.2.0 repeated items already
listed in CHANGES.txt of V5.1.0 -- for details see
https://sourceforge.net/p/oorexx/discussion/408478/thread/e33e7a9161/

---

[bugs:#2078] Interpreter::resourceLock is never locked; static
initialization order clears its created flag

Status: closed
Group: 5.3.0
Created: Sun Jul 26, 2026 09:41 AM UTC by Moritz Hoffmann
Last Updated: Sun Aug 02, 2026 07:13 PM UTC
Owner: Moritz Hoffmann
Attachments:

• provoke_locks.cpp (sourceforge.net)
  (9.4 kB; application/octet-stream)
• provoke_locks.rex (sourceforge.net)
  (524 Bytes; application/octet-stream)

On Unix, Interpreter::resourceLock is never in a usable state, for the
entire
life of every process. Every ResourceSection silently guards nothing.

Mechanism

_rexx_init() in interpreter/platform/unix/SystemInitialization.cpp is
an ELF
constructor. It calls Interpreter::createLocks(), which calls
resourceLock.create(true), setting created = true.

SysMutex Interpreter::resourceLock; is a namespace-scope object in
Interpreter.cpp. Its dynamic initializer, SysMutex() : created(false),
runs
in link order relative to that constructor. In the builds I tested it runs
after, and resets the flag.

SysMutex::request() returns false when !created, so from that point on
every acquire fails and every critical section is unprotected.

Evidence

An ordering probe printing this in the constructor and in
createLocks():

~~~
*** createLocks() created resourceLock at 0x...9e0
*** SysMutex() ctor at 0x...9e0 <- runs afterwards, resets
created
~~~

A probe distinguishing the two failure branches of request() reported
NOT-CREATED for 60 of 60 acquire failures in one test group, at these
call sites:

~~~
InterpreterInstance::poolActivity
InterpreterInstance::terminate (via removeInactiveActivities)
InterpreterInstance::spawnActivity
ActivityManager::createNewActivity
~~~

Interpreter::dispatchLock was declared in the same file and was dead in
the
same way. A probe on the acquire result counts 4855 DispatchSection
constructions that failed to lock in a single GUARD test group run, so the
activity dispatch queue has no protection at all.

Consequences

A deterministic crash. Interpreter::startInterpreter() guards
one-time
bootstrap with:

~~~cpp
ResourceSection lock;
if (!isActive())
{
... memoryObject.initialize(), restore the image, build the local
server ...
}
~~~

With the lock inert, that test is unsynchronised. Several threads calling
RexxCreateInterpreter concurrently all pass it and all restore the image
into
the same global memoryObject. Eight threads creating and terminating
instances segfaults 15 runs out of 15; gdb shows every thread inside
startInterpreter -> MemoryObject::initialize -> restoreImage
simultaneously. A
counter in the bootstrap body records 8 entries where there must be 1.

A test binary for this is attached as provoke_locks.cpp. The rexx
command
line tool creates one instance on one thread, which is why this has not
been
seen in normal use; it needs the embedding C API driven from several
threads.

Data races. ThreadSanitizer reports
InterpreterInstance::allActivities
and ActivityManager::allActivities raced between poolActivity,
removeInactiveActivities and activityEnded, all of which take
ResourceSection in the source. Across six targeted test groups, making
the
lock work takes data races from 45 to 19.

This also explains the unresolved half of bug #1734. After the r12435 fix,
erich_st reported "instead of the intermittent hangs there are intermittent
crashes", with ArrayClass::findSingleIndexItem:2086 <- removeItem:2221 <- poolActivity. That is the same race, and it was not reproducible for
bigrixx
because whether the lock works at all depends on link order.

Suggested fix

Make the lock a function-local static, initialized on first use, so the
ordering cannot invert:

~~~cpp
SysMutex &Interpreter::resourceLock()
{
static SysMutex theLock;
return theLock;
}
~~~

ActivityManager::kernelSemaphore, terminationSem and the dispatch lock
are
namespace-scope objects with the same exposure, safe today only because of
link
order, and want the same treatment.

One thing to be aware of before applying it

With the resource lock actually held, ThreadSanitizer reports a lock order
inversion it could not see before: createInterpreterInstance holds the
resource lock across startInterpreter(), which recurses into
createInterpreterInstance -> getRootActivity() -> lockKernel(),
requesting the
kernel lock while holding the resource lock. Interpreter.hpp forbids
exactly
that, and the comment in createInterpreterInstance requires the
opposite. The
two rules contradict each other and only coexisted because the lock was
inert.

This is the same lock pair as the #1734 hang, and startInterpreter still
has
InstanceBlock inside a ResourceSection — the pattern r12435 removed
from
terminateInterpreter. It is not reachable as a deadlock today, because
active is never reset and so the bootstrap body runs once per process,
before
any other thread can hold the kernel lock. Measured: no hang at 2, 4, 8 or
16
threads. It becomes reachable if startup is ever made re-entrant.

Full test suite with the fix: 24372 tests, 387294 assertions, no new
failures.

---

Sent from sourceforge.net because you indicated interest in <
https://sourceforge.net/p/oorexx/bugs/2078/>

To unsubscribe from further messages, please visit <
https://sourceforge.net/auth/subscriptions/>

[Moritz Hoffmann]

Fixed in r13185 + r13186.

[jfaucher]

Commit [r13185], [r13186]