[Sbcl-devel] Current status for the UDEF-IMMEDIATES contrib

[Philipp M. <ph...@ma...>]

Hi everybody,

here's the current version of my user-defined immediates
contrib.
As the changes in the SBCL core are really minimal now
I'd like to ask for a review resp. inclusion in upstream SBCL.


Summary:

    This patch series provides a column-oriented structure variant
    that's type-safe at compile- and runtime; such objects
    are addressed via immediates with an 8-bit SBCL tag and
    another 8-bit tag to provide multiple such types.

    This reduces memory usage, GC time, and therefore time required
    for safe-lisp-and-die.

The intended use-case is a in-memory database that allocates
lots (>10M) of small structures of a few different types.

The full documentation is available at [1].



This works fine in a production setup,
tested with up to 12 cores parallel allocation.
RAM usage is much better than plain instances,
also because the "pointers" are now mostly (unsigned-byte 32)
or even just (unsigned-byte 8) where possible.

   Class instances:
   - Path 56M instances, deduplicated as 8M fragments using 210MB
   - SHA256 34M instances
   - Version 530k instances, points to Repo, Arch, Dist instances
   In-memory indizes:
   - Versions pointing to 124M (Path, SHA256) tuples
   - SHA256 pointing to 92M (Version, SHA256) tuples

In 2024 the software couldn't import (fewer) data with a 64GB heap;
now, with UDEFs, the heap doesn't grow above ~16GB
and the uncompressed executable is ~7GB.


The contrib directory contains a Makefile
for running a small benchmark script. With
   # make comparison
only allocation is tested, adding an argument
   # make comparison dumpfile=/tmp/text.heap
writes a heap dump as well.
User/System/Wall clock times and max. RAM usage are printed;
while allocation alone is slower (6.3 vs. 5.3 seconds),
the GC on save-lisp-and-die makes the UDEF version faster
(8 instead of 17 seconds); and heap usage is also only half as much.


Known points that could be fixed/enhanced:
- udef subtypes for TYPE-OF
- defmethod with (EQL x)
- accessor performance
- There's no GC (yet); not needed for my usecases,
   full GC could build a freelist for each tag.
- I add a few transforms "by hand", as the specifiers
   collide with built-in DEFTRANSFORMS.
   Having a separate value to differentiate would be great,
   eg. "optimize" for the built-ins vs. "udef" for these here.


Any feedback is welcome, of course.



Ad 1: 
https://github.com/phmarek/sbcl/blob/udef-direct-expansion/contrib/sb-udef-immediate/README.md