64-bit_OWLNext

64-bit OWLNext

This article describes the changes made in OWLNext 6.40 and later to support the 64-bit build mode.

• 64-bit OWLNext
  • Preliminary support in earlier versions
  • Changes in OWLNext 6.40 to support 64-bit
    • Implementation of x86-64 thunks for the supported compilers
    • Replacement of incompatible 32-bit types and Windows API calls
      • A note about handles and truncation
    • 64-bit compatible message dispatch
      • Decoding
      • Encoding
    • Policy for handling 64-bit quantities
  • References

---

Preliminary support in earlier versions

OWLNext 6.32 introduced preliminary support for 64-bit. This consisted mainly of changes to some 32-bit API calls, and an overhaul of the thunking mechanism for message dispatch. OWLNext 6.34 included further fixes to the dispatch framework. However, the 64-bit build mode for these early versions is not officially supported and should not be used in production code.

---

Changes in OWLNext 6.40 to support 64-bit

The following changes have been implemented for 64-bit support in OWLNext.

1. Implementation of x86-64 thunks for the supported 64-bit compilers.
2. Replacement of 32-bit types and Windows API calls by 64-bit compatible versions.
3. 64-bit-correct message dispatch.
4. Policy for handling 64-bit quantities.
5. Build system support for the 64-bit compilers.

Implementation of x86-64 thunks for the supported compilers

This change was implemented in [r759] and shipped in OWLNext 6.32. It involved a complete overhaul of the thunk generation code, since the 64-bit Microsoft compiler no longer supports in-line assembly code. The new implementation creates a thunk by linking a chunk of raw machine code with the window pointer and the address of the dispatch function. This run-time linking is done using a C++ template, which is instantiated for the relevant compiler and build mode. As before the thunk is allocated on an executable memory heap, as to support the Data Execution Prevention safety feature in Windows XP and later.

For more information about thunking in OWLNext, read Windows Message Dispatch in OWLNext, and for the gory details, see the source code (source/owlcore/owl.cpp (sourceforge.net)).

Replacement of incompatible 32-bit types and Windows API calls

To support 64-bit, incompatible 32-bit types and calls had to be replaced by agnostic versions that map to the correct implementation depending on the build mode. This involved the replacement of functions such as SetWindowLong by the agnostic version SetWindowLongPtr. The latter function maps to the former in 32-bit build mode. See forum topic "64-bit Version" for details.

Here is a list of changes made:

• Added TWindow::GetWindowLongPtr and SetWindowLongPtr, and replaced usage of GetWindowLong and SetWindowLong [r756].
• Fixed TResId casting issues [r757].
• Fixed TColor constructor overload resolution problems [r764] [r765] [r775] [r789] [r792] [r1010] [r1026] [r1793] [r1800]. Also see [bugs:#237].
• Defined OWL_BREAK as 'assert' for 64-bit builds [r767] [r799].
• Excluded StackWalker in 64-bit builds [r770].
• Corrected the function return values of TDialog::SendDlgItemMsg, TWindow::SendNotification and SetTimer [r771].
• Updated the signature of LoUint8, HiUint8, LoUint16, etc. for 64-bit compatibility [r781].
• Corrected the signature of TDialog::SetMsgResult [r804].
• Corrected 32-bit truncation in TMDIChild::ShowWindow in call to HandleMessage [r806].
• Corrected 32-bit truncation in TWindow::DefaultProcessing in initialisation of hWndCtl [r807].
• Circumvented the 32-bit parameter issue for virtual TDialog::PerformCreate [r823].
• Corrected the signature of TCoolGrid::EvEndEdit [r854].
• Corrected signatures and data types for TListBox, TComboBox, TListViewCtrl, TTreeViewCtrl (and auxiliary classes, TListBoxData etc.), as well as OwlExt::TCBItemData and TExpandableComboBox [r846] [r852] [r865].
• Corrected the signature of the TEditStream constructor, and corrected narrowing casts in TRichEdit::ReadFromStream and WriteToStream [r868].
• Corrected the type of TGadgetWindow::TimerID [r880].
• Corrected the type of TSocketError::SizeToAllocate [r888].
• Corrected the type of local variable in TMciHiddenWindow::MciNotify [r888].
• Corrected narrowing casts to socket type in TSocket and TStreamSocket [r888].
• Added support for streaming int64 and uint64 in "include/owl/objstrm.h" [r1011].
• Corrected the return type of TDialog::DialogFunction [r1242] [r1260].
• Corrected the type of TDialogAttr::Param [r1709].
• Corrected conversions of TWindowAttr::Param in TMDIClient constructor and in TWindow::Streamer::Read and Write [r1714].
• Changed TNoteTabItem::ClientData from uint32 to INT_PTR to allow pointers to objects to be safely stored [r1894].
• Changed the type of the generic parameter for TDocument::NotifyViews and QueryViews from long to TParam2 [r2598].

Also see the bug tracker for 64-bit issues.

A note about handles and truncation

The Windows API widely uses handles; internal object references (indices) that are cast to pointers, and hence look like regular pointers. While these pointers are 64-bit in x64 build mode, handles only use the lower 32 bits, i.e. the upper 32 bits are insignificant. The MSDN documentation states:

64-bit versions of Windows use 32-bit handles for interoperability. When sharing a handle between 32-bit and 64-bit applications, only the lower 32 bits are significant, so it is safe to truncate the handle (when passing it from 64-bit to 32-bit) or sign-extend the handle (when passing it from 32-bit to 64-bit).

See Programming Guide for 64-bit Windows.

While this means that the handle truncation issues identified above do not matter in practice, fixing them provides C++ type-safety and eliminates suspect code and the related warnings.

64-bit compatible message dispatch

Message dispatch involves the encoding and decoding of message arguments. For 64-bit compatibility all of the message passing code needed to be reviewed and corrected.

Decoding

Version 6.40 introduces a thoroughly overhauled dispatch framework [r1803] [r1804] [r1805] [r1810] [r1811] [r1823] [r1824]. This overhaul improves 64-bit support and simplifies review, testing and maintenance. The old Borland solution was not C++ conformant and was difficult to review and test for correctness. For more information about the old machinery, its issues and its new replacement, see Windows Message Dispatch in OWLNext.

Encoding

Message passing code across all of OWLNext does manual encoding of the message arguments. All of these individual cases have been identified and reviewed for 64-bit correctness. No issues were found (see forum post "64-bit Version").

Policy for handling 64-bit quantities

Proper handling of quantities, such as indices and sizes, that are larger than INT_MAX (~2 GB), is important for robust 64-bit applications. For example, 64-bit quantities arise for memory allocations, standard strings (owl::tstring) and containers (e.g. std::vector), and for file sizes and positions.

The policy adopted for OWLNext 6.40 is to not support 64-bit quantities unless explicitly documented for each function and class. The use of a 64-bit quantity will in general cause undefined behaviour (UB).

This means that, in general, the client code is responsible for detecting and handling 64-bit quantities before calling OWLNext code, e.g. by reporting errors. For example, the client code should check strings for 64-bit size and handle it appropriately, e.g. throw an exception, before passing a string to OWLNext.

void SetText(TEdit& c, const tstring& t)
{
  if (t.size() > INT_MAX) throw runtime_error("Too much text for OWLNext");
  c.SetText(t);
}

---

References

• Windows Message Dispatch in OWLNext
• 64-bit feature request ticket
• 64-bit bug tickets

[Vidar Hasfjord]

Proper handling of 64-bit sizes

Proper handling of 64-bit sizes is important for robust 64-bit support. For example, 64-bit sizes arise for memory allocations, standard strings (owl::tstring) and containers (e.g. std::vector), and for file sizes and positions. We have the following options for the handling of 64-bit sizes, sorted by increasing robustness.

1. Ignore the issue, and document that 64-bit sizes are not supported.
2. Silently clamp 64-bit sizes to [INT_MIN, INT_MAX].
3. Support 64-bit sizes where applicable, and assert (in debug builds) where not.
4. Support 64-bit sizes where applicable, and throw exceptions where not.

For example, many Windows API functions that deal with text, only support a 32-bit size. However, OWLNext uses owl::tstring (and raw arrays) that have 64-bit size. If we ignore this issue, i.e. option (1), sizes will be truncated and cause undefined behaviour, e.g. crashes. Note that truncating a size is not the same as truncating a text. Truncating a 64-bit value gives an arbitrary value in the 32-bit range. This makes option (1) brittle with potentially unexpected results. Option (2), clamping the size to the 32-bit range, has the drawback that it truncates the text. Text truncation is rarely insignificant. Option (3) is unsatisfactory, since the text size is most often determined at run-time, so the issue can not be eliminated in debug builds (asserts should only be used to test programming logic errors). That leaves option (4) as the most robust solution for text.

A similar analysis can be applied to containers and files.

Due to the current state of the OWLNext project, the simplest solution is to adopt option (1), i.e. declare that using 64-bit sizes with OWLNext causes undefined behaviour (UB), unless 64-bit support (or behaviour) is explicitly documented for each function and class. In the long term, option (4) would be desirable.