BaseRT 0.3.0

basert serve is a new server. It keeps the command line and the OpenAI-compatible API it has always had, and runs them on superfluid, our serving daemon: requests are decoded together through one shared forward pass, a prompt a conversation repeats is reused instead of read again, and hybrid models (Qwen 3.5, 3.6 and 3.8) get context windows up to four times larger from the same memory. This release also carries the fixes prepared for 0.2.7, which was never published.

New

• Serving. Continuous batching and prefix reuse are always on; the server runs one request at a time unless --continuous-batching [N] or --max-batch-size N asks for more. A server is ready sooner (Qwen3-8B on an M5 Pro: 5.3 s to 1.0 s). A streamed tool call arrives as partial deltas — the function's name first, then its arguments as they are produced — and tool_choice naming a function, or required, is honoured. /v1/completions accepts suffix for fill-in-the-middle. /v1/models lists each model's capabilities, including whether its template takes a thinking switch and which reasoning-effort levels it defines.
• Larger windows for hybrid models. The paged KV cache charges only the attention layers, so on a 48 GB M5 Pro the automatically sized window grows from 72,704 to 262,144 tokens on Qwen3.6-35B-A3B and Qwen3.5-35B-A3B, from 31,744 to 130,048 on Qwen3.6-27B, and from 27,648 to 111,616 on Qwen3.8-27B. Dense models keep the window they had.
• GLM 5.2 on Metal serves with continuous batching, and speculates with the multi-token-prediction head its bundle carries.
• Speculation. Block drafters with a reduced draft vocabulary (the DFlash heads published for Qwen3-30B-A3B and Nemotron-3-Super) convert and serve, token-for-token identical to plain decoding under greedy sampling. A sampled round verifies all of its rows on the GPU in one pass (an 8-lane round on Qwen3.8-27B: 109 ms to 93 ms). A strategy that loses to plain decoding is measured against a real plain run and retired, then rechecked later rather than written off. basert complete --spec turns speculation on for a bundle with its own head.
• Throughput under agent load. Sampled lanes no longer sort the whole vocabulary per token, served lanes sample on the GPU, and prompts are read prefill-first. Eight agents across six turns on Qwen3.6-35B-A3B (M5 Max): wall time 345 s to 136 s, first token at the median 46 s to 6 s.
• Automatic context sizing on Metal plans within the memory the engine can keep pinned, and counts the recurrent state a hybrid model allocates per lane, so a derived window no longer pushes the weights out during a long prompt.

Changed

• Flags the old server took but this one has no use for are accepted and noted on stderr: --paged-kv and --prefix-cache (always on), --prefix-cache-file and --prefix-cache-save-interval (the cache is not kept on disk), --files-dir and --media-dir (files live in the server's state directory; images and audio arrive as data: URIs or uploads), and the engine settings --metallib, --prefill-chunk, --paged-weights, --no-paged-weights-retry, --gpu-wait-timeout-ms, --decode-replay and --no-baked-decode. --request-timeout bounds fill-in-the-middle completions only. A flag value that is not a number is refused rather than read as 0. --model-dir alone loads its first model at startup. --verbose prints the load and GPU diagnostics without the per-kernel dispatch list.
• State — the request log and uploaded files — lives under $BASERT_STATE_DIR (default: serve/ beside the model cache), one directory per port, or wherever --sessions <dir> says; the request log starts fresh at each start.
• /props no longer reports trained_context; /v1/models reports the serving window as meta.n_ctx; replies say system_fingerprint: superfluid; /metrics series are named superfluid_*. Error messages are worded differently. Without --api-key on a non-loopback address, the server answers only requests addressed to an IP or to this machine's own name.
• Requests that omit sampling parameters keep the old server's defaults (temperature 0, repeat penalty 1.05, and top-p 0.9 / top-k 40 unless the model publishes its own truncation); an explicit value always wins. presence_penalty and frequency_penalty outside [-2, 2] are refused, as before.
• A model cache without hub.json is served under its repository name, and a second variant of the same model as name:variant.
• Text that looks like a chat marker — <|im_end|>, <|im_start|>, <think> — inside a message, a tool result or a document is content, never framing: a quoted marker can no longer end a turn, forge a system turn, or put the rest of a session into reasoning.
• A tokenizer-only handle opens the bundle's metadata only. On NVIDIA that is tens of gigabytes less resident for a server that keeps several.

Fixed

• Gemma 4 26B-A4B misread every image it was shown, on Mac and NVIDIA: a step of its vision encoder was skipped. On NVIDIA its per-head normalization also ran on a partial warp. Gemma 4 E2B and E4B were not affected.
• Hybrid-model prefix reuse now survives a replayed or branched conversation. A conversation replayed after the snapshot store had cycled, or a second turn branched off a shared first turn, read its whole prompt again, so identical loads ran two to three times slower than distinct ones.
• Hybrid models serve at one lane (--max-batch 1), as the Local app starts large ones; the recurrent state the single request needs is now reserved.
• The unfused attention path for 512-wide heads with Q8 KV on Metal misread part of its input on every odd-length prompt.
• GLM-DSA served a stale key row on every single-sequence paged step.
• Llama-family models produced garbage in batch-invariant mode (BASERT_BATCH_INVARIANT=1); Gemma 4 and Qwen 3.5 now use the same kernels in that mode whether a request runs alone or in a batch.
• A LoRA adapter on Qwen 3.5's shared-expert gate is applied during generation on NVIDIA, not only while reading the prompt.
• Two mixture-of-experts paths on NVIDIA no longer misread weights when a model mixes scale formats or keeps routing weights in full precision.
• The GPU sampler treated top_k: 0 as greedy and applied top-p and min-p without the requested temperature.
• A small GPU buffer is freed when a model is unloaded.