|
From: Philipp M. <ph...@ma...> - 2026-06-08 18:15:18
|
Hi everybody,
here's the current version of my user-defined immediates
contrib.
As the changes in the SBCL core are really minimal now
I'd like to ask for a review resp. inclusion in upstream SBCL.
Summary:
This patch series provides a column-oriented structure variant
that's type-safe at compile- and runtime; such objects
are addressed via immediates with an 8-bit SBCL tag and
another 8-bit tag to provide multiple such types.
This reduces memory usage, GC time, and therefore time required
for safe-lisp-and-die.
The intended use-case is a in-memory database that allocates
lots (>10M) of small structures of a few different types.
The full documentation is available at [1].
This works fine in a production setup,
tested with up to 12 cores parallel allocation.
RAM usage is much better than plain instances,
also because the "pointers" are now mostly (unsigned-byte 32)
or even just (unsigned-byte 8) where possible.
Class instances:
- Path 56M instances, deduplicated as 8M fragments using 210MB
- SHA256 34M instances
- Version 530k instances, points to Repo, Arch, Dist instances
In-memory indizes:
- Versions pointing to 124M (Path, SHA256) tuples
- SHA256 pointing to 92M (Version, SHA256) tuples
In 2024 the software couldn't import (fewer) data with a 64GB heap;
now, with UDEFs, the heap doesn't grow above ~16GB
and the uncompressed executable is ~7GB.
The contrib directory contains a Makefile
for running a small benchmark script. With
# make comparison
only allocation is tested, adding an argument
# make comparison dumpfile=/tmp/text.heap
writes a heap dump as well.
User/System/Wall clock times and max. RAM usage are printed;
while allocation alone is slower (6.3 vs. 5.3 seconds),
the GC on save-lisp-and-die makes the UDEF version faster
(8 instead of 17 seconds); and heap usage is also only half as much.
Known points that could be fixed/enhanced:
- udef subtypes for TYPE-OF
- defmethod with (EQL x)
- accessor performance
- There's no GC (yet); not needed for my usecases,
full GC could build a freelist for each tag.
- I add a few transforms "by hand", as the specifiers
collide with built-in DEFTRANSFORMS.
Having a separate value to differentiate would be great,
eg. "optimize" for the built-ins vs. "udef" for these here.
Any feedback is welcome, of course.
Ad 1:
https://github.com/phmarek/sbcl/blob/udef-direct-expansion/contrib/sb-udef-immediate/README.md
|