#1650 Hang on Ubuntu Linux with r11909

[Rony G. Flatscher]

Running the BSF4ooRexx unit tests (using the ooRexx testing framework) revealed a problem(hang) in ooRexx r11909 on 64-bit Ubuntu Linux. The test uses the "-R" option, such that a sequence of testGroups get executed. Executing the testGroup unit test that hangs individually works successfully.

All the same BSF4ooRexx unit tests run successfully on Windows and MacOS with the same version of ooRexx and BSF4ooRexx, so it seems to be confined to Linux.

Enclosed please find the stack trace in normal and full.

[Josep Maria Blasco]

Rony, I think this one may be fixed by Moritz's changes of last August
([r13183]–[r13186], [#2072], [#2074], [#2078]).

Your backtrace shows a lost dispatch: thread 1 is running an uninit
(checkUninitQueue → runUninits) and gives up control
(relinquishIfNeeded → addWaitingActivity → waitForDispatch), while thread 2,
a newly created activity, waits in waitForRunPermission forever, so nobody
lets it run. That's the dispatching code that r13183 changed: several of its
fields (sentinel, waitingAccess, dispatchPosted...) were volatile, which
doesn't order anything, and are now std::atomic.

Without BSF4ooRexx I can't rerun your tests, but I tried the same pattern in
plain Rexx: 1,000 short-lived threads (START) while objects with an UNINIT
method are being collected (about 61,000 uninits per run), with several runs
in parallel. On Linux x86_64:

• [r13182] (just before Moritz's changes): 45 of 60 runs crashed (43 SIGSEGV,
  2 SIGABRT);
• trunk ([r13381]): 0 of 360.

I didn't get the hang itself, not even with [r13182], so this is not a proof.
Could you run your BSF4ooRexx unit tests with "-R" on Linux with a current
trunk build? If they don't hang anymore, I'd close this.

[Rony G. Flatscher]

Dear Josep Maria, unfortunately, I have no working Linux installation currently at my disposal. Maybe Claude can run the tests, after installing Java and BSF4ooRexx850 on its installation? That might be the fastest path to test this. (Otherwise I will try hard to get a Linux machine up and running, but this will be in about two weeks the earliest, given my current overload, I am afraid.)

[Josep Maria Blasco]

Rony, I take back what I said this morning: the hang is still there.

I ran your BSF4ooRexx 850 unit tests (BSF4ooRexx_install_v850-20261103-refresh,
BSF4ooRexx_testUnits.zip) on Linux x86_64 (Ubuntu 24.04, OpenJDK 21) with
ooRexx trunk [r13396], installed in /usr/local, with
rexx testOORexx.rex -R bsf4rexx -s. A run takes 14 to 25 seconds; 7 of 245
runs hung, in different testGroups (09_bsf_compile,
11_box_boxStrictArg_with_Java_class, 15_rexx_NestedClasses).

The six hangs I caught with gdb are identical (backtrace of one attached):

• the "RexxCleanupRef." Java thread is terminating an interpreter instance
  (jniRexxTerminateInterpreterInstance → Terminate → runUninits); an uninit
  method reads a guarded attribute and waits for the object's guard
  (AttributeGetterCode::run → Activity::waitReserve);
• the main thread is in the native BSF() routine, which calls NewString (once
  ArraySize), and waits for the kernel lock (requestApiAccess →
  waitForKernel).

Nothing else is running in the interpreter. It's not the same stack as your
2019 one, but it's the same kind of thing: an uninit and another thread
that wait for each other. I'm looking at who holds the kernel lock and will
report back.

[Josep Maria Blasco]

Found it. The hang comes from InterpreterInstance::terminate() when it runs on
a thread that is not attached to the instance, as BSF4ooRexx's
"RexxCleanupRef." Java thread is.

terminate() calls enterOnCurrentThread(), which gets a new activity from
attachThread(). attachThread() returns with the kernel lock held and the new
activity set as the current one, and then enterOnCurrentThread() requests
the lock again. The kernel mutex is recursive, so the second request
succeeds, and the uninit methods run with it locked twice (I checked
__count under gdb: 2 on an unattached thread, 1 on an attached one).
When an uninit method waits for a guard, Activity::waitReserve() calls
releaseAccess(), which unlocks once and sets currentActivity to NULL: the
thread still owns the kernel while it waits. If the thread that holds that
guard needs the kernel, as the main thread did in BSF() → NewString(), both
wait forever. All 15 hangs I caught had exactly this shape: kernel owner =
the cleaner thread, currentActivity = NULL.

The extra lock was already known: terminate() released it a second time at
the end (made conditional in [r13042], "rebalance terminate kernel locks").
That balanced the count once terminate() was done, but not while the
uninit methods ran.

The fix, committed in [r13400] (code and the test binary) and [r13401]
(test), requests the lock in enterOnCurrentThread() only for an existing
activity, and drops the second release at the end of terminate() (it would
now unlock a free mutex); both cases just nudge the dispatch queue, as the
attached-thread case already did.

A test needs neither BSF nor Java: case 6 of the test binary
detach_root_terminate creates an instance whose object has an uninit method
calling a guarded method of an object of another instance, and calls
Terminate() on a fresh thread while that object's guarded method sleeps in
SysSleep. New test TestTerminateOnOtherThreadUninitWaitsForGuard in
ProcessInvocation.testGroup.

Results:

	without the patch	with the patch
BSF4ooRexx 850 unit tests, -R bsf4rexx, Linux	15 hangs in 561 runs	0 hangs in 200 runs
detach_root_terminate case 6, Linux	hangs 5 of 5 (watchdog)	passes 20 of 20
full test suite, Linux	24,394 tests, the usual 9 failures + the new test	the usual 9
full test suite, Windows (GitHub Actions, MSVC)	24,647 tests, the usual 7 + the new test (*)	the usual 7

(*) On Windows the unpatched run fails the new test because its test binary
has no case 6 yet, so it does not show whether Windows hangs too; the code
is the same and the Windows kernel lock is recursive as well.

The other BSF4ooRexx test failures (29 or 30, and 1 or 2 errors, mostly in
jdor) are the same with and without the patch.