BaseRT 0.2.4

Three new model families, and a context window that sizes itself. gpt-oss, Nemotron 3 Nano and GLM 5.2 all run now, with continuous batching rather than one request at a time. And basert serve no longer starts every model at 4096 tokens — it measures the machine and gives the model the window the hardware can hold, up to whatever it was trained for.

Added

The context window sizes itself

--max-context defaulted to 4096, which threw away most of a 32k model on a workstation and overran the budget on a laptop when raised by hand. It is now derived from the GPU memory available, the weights, and what a token of KV cache costs — capped only by the model's trained window.

Context window: 131072 tokens (auto-sized for 96 GB of GPU memory at 8 lanes).
Pass --max-context N to pin a different one.

--max-tokens follows: unset, a request may fill the rest of the window instead of stopping at 2048. basert chat and basert complete size themselves the same way.

On a Mac sharing its GPU with another program, pin --max-context — the measurement can't see another process's allocation.

gpt-oss

basecompute/gpt-oss-20b in bf16, Q8 and Q4. The MXFP4 experts are carried across verbatim, never requantized. Harmony tool calls are parsed and emitted as tool calls.

Nemotron 3 Nano

basecompute/NVIDIA-Nemotron-3-Nano-30B-A3B in bf16, Q8 and Q4 — a Mamba-2 hybrid, where most layers carry recurrent state instead of a KV cache.

GLM 5.2

Multi-head latent attention, with the sparse-indexer path for long contexts. Continuous batching for it is Metal-only in this release; on CUDA it runs one sequence at a time.

Fixed

• gpt-oss returned NaN on roughly one prompt position in twenty.
• Large models could emit a stray end-of-turn token before answering, on Metal under continuous batching.
• A loaded LoRA adapter ignored repetition, presence and frequency penalties, logit_bias, and seed.
• gpt-oss could keep generating past a finished tool call.
• A failed conversion left a model-sized temporary file behind.
• Image requests overreported prompt token usage when prompt caching was on.
• --validate could not pass on a current MLX checkpoint.

Changed

• BaseRTModelConfig changed layout — recompile anything built against 0.2.3 that reads it.
• The shared library reports its real version again, and the macOS archive now ships the symlink its install name needs. Linking against the library from 0.2.1–0.2.3 failed to load; the bundled tools were unaffected.