[Emc-developers] Moving HAL from 32-bit to 64-bit - status

[Bertho S. <lc...@va...>]

Hello All,

The future upcoming 2.10 release will be seeing quite a few changes. 
Some of them will real breaking changes, requiring you to update 
components, drivers, INI-files and VCPs. The changes are necessary to 
improve both operation and the code base.

The goal of the current changes are: Replacing 32-bit HAL 
pins/params/signals with 64-bit and dropping 32-bit pins/params/signal 
completely.

The goal sounds simple, but there are many technical details that needed 
to be solved before a viable progression plan could be established. For 
discussions of how we got here you can start with #3286 for the 
background and #4099 for the start of the implementation. Fair warning: 
these are very lengthy discussions with a tremendous amount of 
information and detail.

What has happened and will be happening:
1. Replacing direct HAL pins/params/signals access and moving to 
getter/setter
Moving all pins/params/signals to a larger size has consequences 
throughout the code. Many places have hard-coded assumptions about sizes 
and require manual inspection and conversion.
To ease the transition, it was decided to move all access to pins and 
parameters to use a getter/setter approach. Such system hides the 
underlying access to shared memory and protects it in an abstraction 
layer without loss of performance. The user only needs to know that the 
data is accessed properly, not how the underlying system works. The 
second point in the getter/setter approach was to ensure both a smooth 
transition and the continued proper functionality with 32-bit compatibility.
An advantage of the getter/setter change has been that both pins and 
parameters now have a similar structure and use the same access methods. 
This change did expose several bugs where parameters were used wrongly 
or inappropriately.
The current status is that most code has been converted. Only waiting on 
the hostmot2 update. Everything else has been updated, with most of you 
never noticing the change. However, the next step will be a) deprecate 
the old HAL API and b) break the HAL API by removing the old API and use 
the new one exclusively.

2. HAL library isolation and better abstraction
The HAL library internals were being used by other code that should have 
accessed using a defined user-facing API. As with the INI-file update, 
the HAL API update was required to unify, standardize and abstract. All 
of HAL's internals have now been isolated and a new query API has been 
established for all needs to get information about HAL, pins, params, 
signals, components, functions and threads. The last part is to update 
halmodule to add the query API (in the pipeline). The next step is a) to 
deprecate the old access modalities and b) break the API by removing 
access to HAL's internal API completely.

3. Components now requiring a set method to assign values to pins and 
parameters
The way components are build needed to change to enable transitioning 
from 32-bit to 64-bit. There have been created a new set of HAL types 
(HAL_BOOL, HAL_REAL, HAL_SINT, HAL_UINT) that replace the old types. The 
new types are used in halcompile in conjunction with the getter/setter 
system for ease of transitioning.
Also the C/C++ code will see a new set of HAL types:
- hal_bool_t - HAL_BOOL - boolean
- hal_real_t - HAL_REAL - floating point
- hal_sint_t - HAL_SINT - 64-bit (minimum) signed integer
- hal_uint_t - HAL_UINT - 64-bit (minimum) unsigned integer
To accommodate both the old and the new API, we had to add new types to 
halcompile. Changing pin/param type will automatically use the new API. 
The types must change from bit->bool, float->real, s32->si32, u32->ui32, 
s64->sint, u64->uint, unsigned->ui32 and signed->si32. Note that there 
are still 32-bit versions available to halcompile. The 32-bit versions 
are required for compatibility. However, the _underlying_ type is 64-bit 
and the getter/setter infrastructure ensures that it works in a 
compatible way.
The new types in halcompile require a conversion of the component. 
Assigning values to a pin/param can no longer be performed using a 
simple assignment, but is now a macro expanding the setter paradigm. 
That means, 'pin = 123' needs to be rewritten as 'pin_set(123)' and 
'pin++' becomes 'pin_set(pin + 1)'. Please note that adding to pins and 
parameters is an expensive operation. The getter/setter change makes 
that even more clear. [And, no, the getter/setter change does not make 
it slower because the compiler had to perform the exact operation that 
is now written in full.] A PR is in the pipeline (#4256) to perform the 
component update automatically, as much as possible.
All in-tree components and drivers have already been updated, except 
we're waiting for hostmot2. When that is done, then we can a) deprecate 
the old types in halcompile and b) break the API.

4. Making components and drivers work properly in a 64-bit regime
The last issue in the HAL changes is to ensure that all components and 
drivers will continue to work when HAL_S32/HAL_S64 and HAL_U32/HAL_U64 
are all merged into HAL_SINT/HAL_UINT and all the user will see are the 
new 64-bit pin/param/signal types.
This step is the "64-bit clean" step, where code must be reviewed to 
drop inappropriate internal 32-bit truncation in both components and 
drivers. Also supporting code must be reviewed to ensure no truncation 
is happening when it shouldn't.
Then, when the actual merge happens, then the user needs to update 
configurations, code and HAL-files. If they use their own code, then it 
must be reviewed too to be 64-bit clean and interoperable with the rest 
of the system.
Merging types should not really be a breaking step as seen from the code 
internally. However, the user will need to do some work (conversion 
tools may be created by that time). Also, when the old HAL types are 
merged into the new ones, then it may be discovered that some values get 
truncated improperly. These may break some user's setup, although no 
such breakage is expected. It needs some testing time to find the hidden 
32-bit dependencies and properly update them.


We still have some (smaller) changes to go before we start with 
deprecating the old API. There will be a deprecation period before the 
actual API break will happen. Also, some upgrade documentation must be 
prepared when the deprecation hits. The actual deprecation time, until 
we break the API, must still be agreed upon. This should be discussed in 
one of the coming video meetings.

-- 
Greetings Bertho

(disclaimers are disclaimed)