#2109 OLEObject events: returning a value for an out parameter crashes the interpreter; arguments arrive in reverse order

[Josep Maria Blasco]

Description

Two problems in the event methods of an .OLEObject created with "WITHEVENTS", both in OLEObjectEvent::Invoke (extensions/platform/windows/ole/events.cpp). The attached programs show them with ADODB.Recordset, present on every Windows, whose events come synchronously, during the call that raises them. ADO declares

WillMove([in] EventReasonEnum adReason, [in, out] EventStatusEnum* adStatus, [in] _Recordset* pRecordset)

Seen with ooRexx 5.3.0 trunk (REXX-ooRexx_5.3.0(MT)_64-bit 6.06 9 Oct 2026), Windows 11.

1. Returning a value for an out parameter crashes the interpreter

ole-events-crash.rex MODE adds two records with a WillMove method that returns nothing (none), returns 1 (one: adStatusOK, a valid value for adStatus), or only reads its arguments (args):

for %m in (none one args) do @rexx ole-events-crash.rex %m & echo %m: exit code %errorlevel%

none: done, 5 WillMove calls
none: exit code 0
one: exit code -1073741819
args: done, 5 WillMove calls
args: exit code 0

(-1073741819 is 0xC0000005, an access violation. Run it with cmd /v:on and !errorlevel! inside a FOR.)

The method's return value goes to the out parameter through Rexx2Variant(context, rxResult, &pDispParams->rgvarg[i], pDispParams->rgvarg[i].vt, -1). There rgvarg[i] is VT_BYREF | VT_I4, pointing to the caller's storage. Rexx2Variant strips VT_BYREF and, for a number, ends in VariantChangeType(pVariant, &sVariant, 0, VT_I4), which overwrites the caller's VARIANT itself with a VT_I4 by value. The pointer is replaced by the value (1), the caller's storage never changes, and the caller then dereferences what it still takes for its pointer. (VT_BOOL, VT_R8 and VT_R4 are written through the pointer; the other types go through VariantChangeType like this.) A value for an out parameter should be converted into a temporary VARIANT and then stored through V_BYREF(&rgvarg[i]) according to its VT, leaving rgvarg[i] as it was.

Related, not explained yet: with Excel, returning .true from WorkbookBeforeClose(Wb, Cancel) (a VT_BOOL) does not stop the workbook from closing either.

2. The arguments arrive in reverse order

ole-events-repro.rex prints WillMove's arguments on MoveFirst. Expected 12, 1, <the Recordset>; got (from an earlier version of the program, which also printed the Recordset's RecordCount):

WillMove got 3 arguments:
  arg 1: an OLEObject (RecordCount 2)
  arg 2: 1
  arg 3: 12

OLEObjectEvent::Invoke converts pDispParams->rgvarg[i] into argument i + 1. COM passes positional arguments in reverse order (rgvarg[0] is the last one), and named arguments (cNamedArgs, rgdispidNamedArgs) first, so argument k (from 0) is rgvarg[cArgs - 1 - k] when there are no named ones. The same loop looks for out parameters with pList->pusOptFlags[i], in declaration order, and writes into rgvarg[i], in reverse order: with more than one parameter it picks the wrong argument (with ADO's three, the middle one, by chance the right one). Excel's events did arrive in order in our tests, perhaps because Excel passes them as named arguments.

The patch

ole-events.diff (against trunk r13268, events.cpp only):

• Each parameter's VARIANT is found in the event's order: named arguments at their positions (rgdispidNamedArgs), then the positional ones from the end of rgvarg. The arguments, and the out parameters' flags (pusOptFlags, declaration order), use that same order.
• A value for an out parameter is converted into a temporary VARIANT of the target's type and stored through the parameter's pointer (V_BYREF): scalars copied, a BSTR or an interface replacing (and freeing) the old one, a VARIANT replacing the old one. The caller's VARIANT itself is not touched. A value that cannot be converted leaves the parameter unchanged.

ole-events-test.rex checks it with ADO: the arguments in order; returning adStatusOK (no crash); returning adStatusCancel cancels MoveNext; and MoveComplete(adReason, pError, adStatus*, pRecordset), whose out parameter is the third of four, returning adStatusUnwantedEvent stops further MoveComplete events. Built with Visual Studio 2022 (x64) from trunk r13268: unpatched, ole-events-crash.rex one crashes and the arguments come reversed; patched, the three modes finish and the test passes (5 of 5).

[Josep Maria Blasco]

Tried with Excel (Microsoft 365) on Windows 11, ooRexx r13268 with the patched orexxole.dll: returning .true from WorkbookBeforeClose(Wb, Cancel) now keeps the workbook open, and Excel’s arguments still arrive in declaration order. So the patch also fixes the Excel Cancel case mentioned as “not explained yet” in the report.