Semlith Files

Indexing, several times faster on the machine you already have

An index pass is three stages running at once. A pool of threads sized from the performance cores reads, hashes, extracts, scans, parses, chunks and tokenises each file once; one writer commits rows and vectors and never runs a model; an embed stage hands token-budget batches to every device with two in flight on each and lands the vectors in id order. Before this release one thread did all of it and ran the CPU's model inline, which was 64 % of a run while the GPU lane waited for batches. Where the time went — walk, read and hash, extract and scan, parse and chunk, tokenize, the wait on each lane, and write — is in the run card, the daemon's log and semlith index --verbose, and the parts add up to the run's wall time.

The Neural Engine, on Apple silicon. granite re-implemented in Apple's Neural Engine layout and converted to Core ML at fixed shapes places 3 375 of its 3 380 ops on the Neural Engine, and runs at 236.5 chunks/s on the M1 Air against the CPU's 30.7, with every vector at cosine ≥ 0.9999 against the fp32 reference. semlith setup fetches the models (built in CI, pinned by digest); the first start compiles them for the Mac, with its progress and the time left on Machine limits: the lane takes batches 39 s after it starts on the M1, once its longest bucket has compiled, and a later start, an upgrade included, is ready in about 2 s. A run waits for a lane that is on and still downloading, starting or compiling, and its card says what it is waiting for; the CPU carries a run only when it is switched on or no lane is ready or on its way. While the Neural Engine runs, nothing else embeds beside it: two CPU threads cut it to a third, and the GPU beside it adds 4 % sustained on a fanless Air (semlith accel on gpu-beside-ane turns that back on). Without the Neural Engine the GPU runs through Core ML (73.2 chunks/s on the M1, against WebGPU's 43.7). Every vector any lane returns is checked finite and non-zero, and one that is not is embedded again on the CPU.

A machine-wide vector cache. The same chunk under the same model, variant, chunking rules and truncation gets its vector from the cache instead of a device: an edit re-embeds the functions it changed rather than its whole file, and a second worktree re-embeds almost nothing. One file under the semlith home, bounded by a least-recently-used cap (1 024 MB; Machine limits and vector_cache_mb set it, 0 turns it off). stats, semlith_stats, the run card and index --verbose show its size and hit rate. Compaction never touches it.

A store that answers while it fills. Rows are committed as they are written and a pass starts on the files changed most recently — by modification time or by the last commit that touched them, in every repository up to two folders below the one indexed, code before licences within one commit — while the walk of the rest goes on beside it. The writer runs up to sixteen windows ahead of the devices, and the daemon no longer scans every file for a run's plan before starting it. Keyword and graph questions answer about what has been read so far; a semantic question embeds up to sixteen of its best keyword matches that have no vector yet and says how much of the store is still pending.

Everywhere else. semlith index and semlith watch in a terminal use the same lanes as the daemon. WebGPU prefers a discrete GPU over an integrated one (gpu_adapter names one; doctor --gpu says which and why). A bulk index runs at Utility QoS on macOS rather than background, below normal priority with EcoQoS off on Windows, and under Nice=5, CPUWeight=50 and IOWeight=50 from the systemd unit on Linux. The CPU lane runs granite on ONNX Runtime directly with thread spinning off.

Experimental lanes, built without their hardware. NVIDIA's TensorRT for RTX and Intel's OpenVINO, loaded from the vendors' own plugin execution providers, and llama.cpp's server (Metal on macOS, Vulkan on Windows and Linux) on loopback with a key only its worker holds. Each is off until turned on, labelled experimental in accel status, doctor --gpu and the portal, says why it cannot run on a machine without its device, and falls back. No throughput is claimed for any of them. The CUDA lane becomes experimental too; a saved cuda: true stays on.

Fixed

• A store built with one of fastembed's own models no longer has its batches sent to a GPU lane, which runs granite and would have given it vectors from the wrong model.
• semlith doctor --gpu printed n/a for a lane that failed its check; it prints FAIL.
• A pack that fails its digest is deleted whole, not left half-unpacked.
• The GPU lane through Core ML was sent batches of one to three chunks on a model that computes eight at a time, so three quarters of its work was padding, and the lane's pace, read from those batches and charged for the wait of a second batch queued behind the first, cut the next batch further. A lane's batch is now a whole number of the model's calls, and its pace is its own time on the device. With the Neural Engine off, the M1 indexes at 62.7 chunks/s p10 where it indexed at 16.0; the release's gate for that case, 80, is not met (#164).
• The vector cache takes 1.7 KB on disk a vector, where it took 4.7 KB: its table kept about a kilobyte of each row on the row's page and moved the rest of the vector to an overflow page of its own, so a full cache was three times its cap. A lookup takes 8.6 µs, where it took 27. A cache written by an earlier 0.32.0 build keeps its layout; deleting cache/vectors.db rebuilds it.
• semlith stats names every lane that is on, the Neural Engine and the experimental lanes included; it named only the CPU, the GPU and CUDA.
• semlith accel status shows the running daemon's lanes, so a Neural Engine compiling in the daemon no longer reads as idle in the terminal.
• A run's rate counts vectors that have landed, not rows written ahead of them.
• The portal's pages no longer wait seconds on a large store: the Stores figures read how many lines each file spans from a new index on the chunks table instead of every chunk's text (14 s to 0.2 s on a 600 MB store). It is built once when a store is first opened by 0.32.0.
• A daemon stopped while an accelerator lane was restarting no longer waits for the lane: batches queued on a lane that is not ready go to the CPU.
• A killed lane worker no longer takes the daemon down with it.
• The Neural Engine no longer compiles for minutes on every start or fails its start. Three causes. A worker with no batch for a minute was retired even while it was still compiling, so the compile was abandoned half written, the system's compiler went on with it anyway, and the next start compiled it again: a backlog of those once left a lane compiling for over two hours. macOS caches a compile under the program's name, so every new binary compiled all six buckets again. And each start waited for all six. A Core ML worker is now kept until its last bucket has loaded, runs from a copy of semlith kept beside the models, is ready after its longest bucket, compiles the rest at the priority of a user's request, lets one compile run at a time, and deletes the 94 MB a killed compile leaves behind. On the M1: ready 39 s after a cold start with all six compiled 2.5 min later, and ready in 2.1 s on every start after that, an upgrade included, where it had taken 15 min.

Install

macOS and Linux

:::sh
curl -fsSL https://raw.githubusercontent.com/semlith/semlith/main/install.sh | sh

Windows

:::powershell
irm https://raw.githubusercontent.com/semlith/semlith/main/install.ps1 | iex

Or download the archive for your platform below, unpack it, and put semlith on your PATH. Intel macOS has no prebuilt binary: ONNX Runtime no longer publishes x86_64 macOS builds.

Verify a download against SHA256SUMS before running it.