#881 Add Regina's RexxCallback to ooRexx

[Rony G. Flatscher]

Mark Hessling has created many external Rexx function packages from RexxSQL to Rexx/DW. For the function packages that rely on callbacks from native libraries Mark implemented a RexxCallback-API into Regina.
If ooRexx had the Regina RexxCallback-API, then all external Rexx function packages that need that API can be used from ooRexx from that moment on, adding valuable utility to ooRexx allowing users of these APIs to also migrate to ooRexx, if they desire.

[Anonymous]

Summary of that as an item of work.

RexxCallBack in ooRexx: a modest job, about 2–4 days including tests and docs. ooRexx 5.3 doesn't have it (I checked its source). Regina's RexxCallBack(name, argc, argv, &rc, &result) lets an external function call a label in the Rexx program that called it.
The interpreter already does almost all of the work:
NativeActivation::findRexxContext() finds the calling Rexx activation.
RexxActivation::getLabels() / RexxCode::getLabels() look up the label.
RexxActivation::internalCall(name, target, args, argc, result) is exactly what CALL label runs. It handles SIGL, traps and the new activation.

What has to be written:
API (about 20 lines): declare RexxCallBack in rexx.h, export it (the Windows .def file and the Unix symbol list), and give it Regina-compatible return codes.
Implementation (about 100–200 lines, api/):

Take the current activity and the running native activation, then find its Rexx context. If it isn't called from Rexx, return an error code.

Look up the label. If it's missing, return a not-found code, as Regina does.

Convert the RXSTRINGs to RexxStrings, call internalCall under the right activity and lock handling, and turn the result back into an RXSTRING (RexxAllocateMemory).

Map a SYNTAX error or uncaught condition to a return code instead of letting it unwind through C.
Optional: an object-oriented version, context->CallLabel(name, array) for the RexxCallContext API. That means adding an entry at the end of the function table and raising the interface version.
Tests and docs: ooTest cases (argument passing, RESULT, a missing label, SIGNAL out, nested callbacks) and a section in rexxpg.

The risk is in the edge cases, not the call: a callback from a different thread, from an exit handler, or while a condition is pending. The RexxLA reviewers would focus on those.

For rx2cpp: compiled programs would also need a table of internal labels so the runtime can find a label by name (more effort). In mixed programs, a label in interpreted code works as soon as ooRexx has the API.

(Parser rebuild still running; results to follow.)
Architecture = Tom Dyer, Coding = ChatGPT / Claude, Writer = Opus 5.5

---

I would expect that the testing time would be the substantial effort.

Tom

looking at acheduling it now.

[Josep Maria Blasco]

+1

[Josep Maria Blasco]

I have an implementation of RexxCallBack ready, compatible with Regina's
(thanks to Tom, who was looking at this too and was happy for me to go
ahead), and I've attached it as three patches: the interpreter, a new test
group and the documentation.

It works as in Regina. An external function calls a label of the program
that called it, as an internal function call from that clause: same
variables unless the label uses PROCEDURE, SIGL set, NUMERIC settings and
traps inherited and restored. The return codes are Regina's. An error in
the called label, or an EXIT, ends the external function and is raised at
the clause that called it, which is also what Regina does, and what an
event loop needs: otherwise a GUI program would hang on an error in a
callback. Called from a thread that is not running Rexx, RexxCallBack
returns RX_CB_NOTSTARTED and never starts an interpreter instance.

The changes are small: the declaration in rexx.h (with the REXXCALLBACK
macro, which packages test to turn their callbacks on), the export in
rexx.def, the call itself in InterpreterAPI.cpp and NativeActivation.cpp,
and four classic test functions in orxclassic1. The new test group,
ooRexx/API/classic/RexxCallBack.testGroup, has 19 tests; the full test
suite gives the same results with the patch (Linux x86_64).

I also tried it with real packages written for Regina, on Linux:

• A function package built against Regina's libregina runs unchanged on
  ooRexx, callbacks included.
• Mark Hessling's Rexx/DW 2.1, on Dynamic Windows 3.4 and GTK3: its GUI
  callbacks run as on Regina, both with the binary built for Regina and
  rebuilt for ooRexx (it turns its callbacks on by itself when REXXCALLBACK
  is defined). Without the patch the same program hangs, or does not load.
• Rexx/SQL 2.6 with SQLite3 (it does not use callbacks) runs unchanged too.

One thing for Mark, if he reads this: the configure scripts of Rexx/SQL,
Rexx/DW and Rexx/CURL select the ooRexx 4 types (OOREXX_40) only when
oorexx-config reports a version starting with 4, so with ooRexx 5 they
need -DOOREXX_40 passed by hand. With that, Rexx/SQL builds for ooRexx
without changing a line.

[Rony G. Flatscher]

This is awesome, thank you very much!

[Josep Maria Blasco]

A note on Windows. On Linux, a package built for Regina can be used by
ooRexx as it is, because the dynamic linker resolves its calls to the API
in the ooRexx library that is already loaded. Windows does not allow that
kind of reuse: a DLL built for Regina imports the API functions from
regina.dll by name, so it would keep calling Regina, not ooRexx. On Windows
these packages have to be rebuilt against ooRexx's headers and libraries,
which the patch makes possible without changing their source (the
REXXCALLBACK macro turns their callbacks on). I haven't tested any of this
on Windows: the Windows parts of the patch (the rexx.def entry, and the
thread used by one test function) are written but not built.

[Rony G. Flatscher]

In the past, Mark had developed a package named Rexx/Trans, which might be interesting here, see https://rexxtrans.sourceforge.net/.

[Josep Maria Blasco]

With the current tools, it's practically costless to generate .dlls and .sos for Regina and ooRexx simultaneously -- you simply program it into Continuous Integration and you're done. And you save a layer...

[Josep Maria Blasco]

I've now built and tested the patch on Windows too, with a GitHub Actions
workflow (MSVC, 64-bit; https://github.com/JosepMariaBlasco/oorexx-bugs),
and it found two problems in the patch I uploaded earlier. The attached
rexxcallback-main.patch replaces it:

• The test library did not compile: orxclassic1.c used a variable called
  "small", which the Windows headers define as a macro for char.
• Unbounded recursion through RexxCallBack killed the process instead of
  ending with Error 11.1. With MSVC a catch block runs on top of the frames
  it caught the exception from, so throwing again inside it keeps them all;
  the patch did that at every level. It now throws after the catch block.
  Nothing changes on Linux.

With the new patch, on Windows: the RexxCallBack test group passes (19
tests), and the full test suite (24,431 tests) shows the same 7 failures as
an unpatched build of the same revision (the printer tests, which need a
printer, two tests of output redirected to a file, and TEST_ARG_LIMIT in
SysGetXxxPathName). On Linux, the same results as before.

One difference between the platforms, now measured: on Linux, a package
built for Regina can be used by ooRexx as it is, because the dynamic linker
resolves its calls to the API in the ooRexx library that is already loaded.
Windows does not allow that kind of reuse: a DLL built for Regina imports
the API from regina.dll by name. Mark's Rexx/SQL DLL built for Regina
loads, but its SQLLoadFuncs registers the functions with Regina, so ooRexx
then says it cannot find SQLVARIABLE. On Windows these packages have to be
rebuilt against ooRexx, and that works without changing their source:
Rexx/SQL 2.6 built with its own vcwin64.mak (INT=OOREXX4 DB=LITE, SQLite
from vcpkg) runs the same test as on Linux and on Regina, with the same
results. Mark's Rexx/SQL DLL built for ooRexx 4 also runs unchanged on
ooRexx 5.

[Josep Maria Blasco]

Committed: [r13307] (interpreter: rexx.h, InterpreterAPI.cpp,
NativeActivation, rexx.def, and the classic test functions in
orxclassic1), [r13308] (tests: ooRexx/API/classic/RexxCallBack.testGroup,
19 tests) and [r13309] (docs: RexxCallBack in the rexxapi book).

The code is v2 of the main patch (the two Windows fixes) with the test and
docs patches as posted. Revalidated on [r13306]: the test group 19/19 on
Linux and Windows, and the full test suite shows the same results as
without the patch, apart from the new group.

[Mark Hessling]

Hi Josep Maria,

This is exciting news!! Many thanks for this. I take note of the OOREXX_40 issues and had already resolved this is my build system. I'll have to do some investigation into the statement you made that a Regina-built .so, like librexxsql.so will work with Regina or ooRexx at the same time. This will have a huge impact on my packaging of all of my external function packages.

Unfortunately I haven't seen anything run yet with ooRexx's RexxCallback() as I get an error building the latest ooRexx SVN code.

My steps:
svn checkout ... into ~/src/oorexx-code
cd ~/oorexx-build
cmake ~/src/oorexx-code

Error:
[ 74%] Generating lib/rexx.img
103 - ::method environments class external 'LIBRARY REXX cmdhandler_environments'
124 - call 'CommandClasses.orx' rexxPackage -- command environments written in Rexx
Error 90 running /home/mark/src/oorexx-build/bin/CommandClasses.orx line 103: External name not found.
Error 90.998: Unable to find external method "cmdhandler_environments".
Logic error: Error building kernel image. Image not saved.
Illegal instruction (core dumped)
make[2]: *** [CMakeFiles/rexx_img.dir/build.make:78: lib/rexx.img] Error 132
make[1]: *** [CMakeFiles/Makefile2:453: CMakeFiles/rexx_img.dir/all] Error 2
make: *** [Makefile:156: all] Error 2

Cheers, Mark

[Josep Maria Blasco]

Hi Mark,

Glad it's useful! About the build error: trunk is fine, but I think your rexximage is loading an older librexx.so than the one it was just built with. cmdhandler_environments only arrived in r13288 (three days ago), and the build-tree binaries find librexx through RUNPATH, which LD_LIBRARY_PATH overrides.

I could reproduce your output line by line: trunk r13312 builds cleanly here, but with LD_LIBRARY_PATH pointing at a librexx built from r13287, make stops at rexx.img with exactly your Error 90.998, the "Logic error", the illegal instruction and make's Error 132. Could you check these in your build directory?

echo $LD_LIBRARY_PATH
ldd bin/rexximage

If ldd shows a librexx.so outside the build tree, this should get you through:

env -u LD_LIBRARY_PATH make

About the Regina-built .so: I should have been more precise. On Linux it works because the package's API calls resolve to the interpreter library already loaded in the process, but a .so linked with -lregina still has libregina.so.3 as a dependency, so Regina must be installed for it to load under ooRexx, even though it isn't used.

I tried the alternative: Rexx/SQL 2.6B3's objects linked without -lregina (just -lsqlite3 -lm) give a librexxsql.so with no interpreter dependency at all, and it passes the same test:

• under regina (the API binds to libregina.so.3), and
• under ooRexx 5.3.0 (it binds to librexx.so.4).

If that suits your packaging, one Linux build could serve both interpreters. Windows still needs a separate build for each.

Cheers, Josep Maria