BaseRT 0.2.0

A second hardware backend. BaseRT now runs on NVIDIA GB10 (DGX Spark, Linux/arm64) alongside Apple Silicon — the same .base bundles, the same basert CLI and OpenAI-compatible server, now on CUDA. This release also brings the Qwen3.5 / Qwen3.6 hybrid model family and a continuous-batching overhaul of basert serve.

Added

• CUDA backend — NVIDIA GB10 / DGX Spark — full inference on Linux/arm64 + CUDA: dense, MoE, and hybrid (Gated-DeltaNet) architectures, across prefill, single-stream decode, and continuous-batching serving. install.sh auto-detects the platform (macOS/arm64 → Metal, Linux/arm64 → CUDA) and basert pull resolves the right bundle for the host.
• Qwen3.5 & Qwen3.6 hybrid models — support for the Gated-DeltaNet hybrid architecture (dense and MoE), including vision (Qwen3.5-VL), on both Metal and CUDA. GB10 continuous batching for the dense hybrids.
• Continuous-batching serving — basert serve now drives tool calls (streaming + non-streaming), grammar-constrained decode, per-token logprobs, and n>1 choices through the continuous-batching engine, with a decode-priority scheduler and radix prefix-cache reuse (dense and hybrid, via a keyed GDN snapshot store).
• CUDA-native model bundles — cuda-q8 and cuda-q4mix variants for 11 models (Qwen3-0.6B/30B-A3B, Qwen3.5-2B/35B-A3B, Qwen3.6-27B/35B, Llama-3.2-1B/3B, Gemma-3-1B, Gemma-4-E2B/26B), on Hugging Face and the catalog.
• Faster Metal GEMMs — large-tile prefill GEMM kernels for native bf16 / f16 weights and Q8, plus batched Gated-DeltaNet decode/prefill kernels for the M1–M5 families.
• Tokenizer conformance — an HF-exact conformance harness and a Unicode-category pretokenizer; Phi-3-mini un-quarantined after revalidation.

Fixed

• Chunked-decode correctness on CUDA — free generation past the decode chunk size no longer repeats an earlier block (an eager-dispatch write-after-read hazard); prefill and teacher-forced decode were always correct.
• Large quantized tensors — weight tensors with more than ~512M elements no longer decode to garbage (64-bit byte-offset fix).
• Gemma normalization precision — the canonical post-attention / feed-forward norm weights are kept in floating point in the shipped bundles (were quantized in some profiles).
• Hybrid prefix reuse — correct KV/GDN dtype handling and prefix-cache reuse for hybrid-GDN models under serving.
• MoE quantization parity — Qwen3.5 MoE quant rules matched to canonical tensor names so experts quantize as intended.

Changed

• Backend-aware model resolution — CUDA hosts prefer CUDA-native bundles, Apple Silicon prefers Metal, with a universal fallback; catalog entries are backend-tagged and can be derived from a .base header.
• Performance — Metal decode/prefill tuning across the M1 Max / M4 Pro / M5 Pro families (large-tile GEMM, higher-occupancy Gemma-4 MoE gate/up). Benchmarks vs llama.cpp and vLLM on GB10 and Apple Silicon are in the repo.

Install

:::sh
# macOS (Apple Silicon, Metal) or Linux (arm64, CUDA) — auto-detected
curl -fsSL https://raw.githubusercontent.com/basecompute/baseRT/main/install.sh | sh
basert pull <model>          # resolves the right bundle for your host
basert serve <model>.base    # OpenAI-compatible server