ChrysaLisp 7.0 "Pipsqueak"

The Lisp engine squeezed till the pips squeak, a network that goes quiet when it has nothing to say, and a machine that sizes its own network. Small, and much quicker than it looks.

Every machine on a network must run this release. The routing ping and the shared memory link status have both changed. An install from before this cannot build its way forward, update with make install, which starts from the new snapshot.zip, and rebuild the host programs with make.

The Lisp engine

• 40% fewer instructions over the test suite, and a full build of the system on one node in 0.32 seconds, from 0.54, on an Apple M4. A Raspberry Pi 4 builds in 1.3 to 1.5 seconds, the best seen before was 2.3.

• The recursive core holds a fraction of what it did on the stack. A special form and the body of a lambda hold nothing at all, a call holds 16 bytes while its args evaluate.

• The args list and the environment of a call are reused, not made and freed. A prebound function is not evaluated. The Lisp object is fetched from the task control block, not kept in a stack slot at every level.

• The boot environment is spread over buckets once loaded, a lookup of an unbound symbol was a scan of all 828 of its symbols.

• The :lisp class, all but its init and deinit, is now written with registers.

• New doc, docs/ai_digest/till_the_pips_squeak.md, has every step, what it saved, how it was measured, and what was tried that did not pay.

The network

• The routing ping no longer carries a node's services, only a hash of them, and backs off from 1 second to 64, so a settled network goes quiet. A link coming up or going down kicks it awake.

• A shared memory link knows the process at the other end and asks the host if it still runs. A dead node is known for certain within a second, and a busy one is never taken for dead. A link that times out is marked down, not torn down, and comes back.

• An idle network burns far less, a fully connected 27 nodes went from 3.76% of a core each to 0.85%.

• A node tells its neighbours its load, not just its task count, and a new task goes to the least loaded.

• Every node knows which machine every other node is on, (lisp-nodes :t), (lisp-nodes system) and (lisp-systems).

• Mail for a node that is still known is no longer freed as undeliverable.

• New topology launch scripts, ring, cube, tree, mesh and star.

The host

• New host calls, (pii-spawn) (pii-pid) (pii-alive) (pii-cpus) and (pii-memory), on Darwin, Linux and Windows.

• A network can size itself. ./run_tui.sh -n 0, and run.sh, start one node for each processor, fully connected. make install uses it. (node-spawn), (node-auto) and nodes -a add nodes to a running network.

• RISC-V 64 is running again, and LoongArch 64 is running, both tested under QEMU with byte identical self hosted boot images.

The assembler and its checks

• Call fusion on the link register targets, boot images 2.4% smaller on ARM64 and 4.2% on RISC-V and LoongArch.

• The generic VP optimiser is one sweep, and a bug in it is fixed, a push or a pop between two reads of a stack slot was not seen to move the stack.

• A validate build, make it validate, now fills cells on alloc and on free, guards the end of each cell, and checks every ref and deref, to catch use before set, use after free, double free and overruns. It found two real faults at once, both fixed.

• trace -i -l lints every function, with no filters.

Tests

• A new test framework, tests/suite.inc, and the suite has grown from 1,629 tests to over 3,300. The tests command prints only failures and a summary, runs modules in parallel over the nodes, and takes -m, -v, -l, -j and -f.

• The new edge case modules found and fixed faults in (nlz), (trim), (join), (swap), (slice), the shifts, (nums-sum), (json-stringify), (map!) and (filter!), (sort), (replace-regex), (tree-load), regexp matching at the end of text, whole word search, editor replace, and the Document and Buffer classes.

Services, commands and apps

• The lock service, @Lock, has shared read and exclusive write locks on hierarchical paths, scoped (with-lock) macros, a history, and a locks command. The file commands, libraries and all the GUI apps use it for their files and configs.

• The task farms no longer lose a job on a slow machine.

• lisp -r runs code given on the command line. New brackets command. The reader reports an unbalanced form, it used to crash the node.

• mail-read-timeout cancels its timer when mail arrives first, 40,000 timed reads went from 5.4 seconds to 15ms.

• Onslaught is feature complete against the C++ original, with the campaign map, load and save, and panned sound. It can be played remotely over its @Onslaught service, and the onslaught command has a bot that plays it.

• New lib/streams/hex.inc, (ui-save), View :add_before and :add_after, large strings in tree-save and tree-load, and tool tips on six more desktop apps.

Where it came from

• New docs/history/history.md, "How We Got Here". The Virtual Processor began in 1989 as a set of assembler macros under two Atari ST and Amiga games, became the Taos operating system, which won the UK's Computing gold award for innovation in 1994, then Tao Group's intent, and is the same idea this system is built on. The press cuttings of the time, from The Independent, BYTE, Edge, the IEE Review and others, are in docs/history/press/, each credited to its author.

Windows

• Installed, built and tested on Windows for this release, with thanks to Martyn Blyss. The TUI and GUI run, the system rebuilds itself to byte identical boot images, a full build takes 0.19 seconds, and the test suite passes.

• The first Windows run found that starting a node took down the node that asked. A host call runs on the stack of the task that makes it, and on Windows CreateProcess needs far more than a task has. (pii-spawn) is now made by the kernel task, on the host's own stack, on every host. nodes -a works there.

Not ready

• The fmt command has a new engine, it is an experiment and is parked. Do not use it on the tree.

• On Windows, -n 0 in the PowerShell scripts has not been run, and the batch files do not take it.

The full notes for every change are in STATUS.md.