#48 fix(local-llm): managed Qwen 3.5 stalls in an unterminated tool-call grammar

[Anonymous]

Originally created by: sachin-detrax

Closes [#47].

Managed mode never returned a response on qwen-3.5-4b (the default managed model) while the same model in external mode worked fine. That split is what isolated it: the difference is the managed daemon's launch flags, not the model or the hardware.

Verified on the reporter's machine (RTX 5070 Ti 16 GB / Ryzen 9 9900X / 32 GB) — managed mode now streams tokens and completes turns.

The bug

models-catalog.ts attached a bundled chat template to qwen-3.5-4b and qwen-3.5-35b, forwarded by startDaemon as --chat-template-file. That file is a Qwen2.5-style ChatML template with no thinking markers:

$ grep -c think assets/ai-models/qwen3.5-chat-template.jinja
0

llama-server echoes the override back at /props.chat_template, which is the only signal detectModelProfile has. Without <think> / enable_thinking, looksLikeQwenThinkModel fails and the profile is demoted to plain-instruct. A/B with the same model and alias, only the template differing:

MANAGED (--chat-template-file bundled)     EXTERNAL (GGUF's own template)
  profile : plain-instruct                   profile : qwen-think
  root    : root ::= tool-call-array         root    : root ::= think-prelude tool-call-array
  1st char allowed: "[" ONLY                 1st char allowed: any (reasoning prelude)

buildGrammar early-returns for a reasoningStyle: "none" profile, so the reasoning prelude leaves the GBNF root. A reasoning model is then forbidden from reasoning: it emits the forced [, lands in the unbounded ws ::= [ \t\n\r]*, and since llama.cpp hard-masks EOG until a grammar stack empties, whitespace is an infinite legal move.

It is silent because assistant_delta is only emitted once the stream parser reaches {"tool":"reply","args":{"text":". Having consumed the [, the parser sits in json_tool matching "tool": against whitespace:

StepEvents emitted after '[' + 4000 newlines: 0

Changes

1. Drop the Qwen 3.5 chat-template override (models-catalog.ts, asset deleted). The GGUF ships the correct template; managed mode is now byte-identical to the working external setup. The chatTemplateAsset mechanism stays as an escape hatch with the invariant documented: an override MUST preserve every reasoning marker profile detection keys on.

2. Bound ws to ( [ \t\n\r] ){0,64} (grammars/tool-call.gbnf). Defence in depth — any future profile/grammar mismatch now fails fast instead of burning 8192 tokens in silence. Same grouped {0,n} shape already proven by the reasoning-seam whitespace rule. 64 is far past any realistic newline+indent run; past that the mask simply forces the next real token.

3. Stop retrying our own requestTimeoutMs abort (llama-server-client.ts). It surfaces as status === null, structurally identical to a dropped socket, so isRetryableLlamaError replayed it up to completionRetries times — 3 × 300 s of silent GPU churn before anything surfaced. LlamaServerError now carries timedOut and short-circuits like a 4xx, with a message naming the knob to raise. Genuine transport failures still retry.

4. Raise MIN_AUTO_CONTEXT 8192 → 16384 (context-size.ts). The old value was exactly equal to the default completionMaxTokens and smaller than the agent's fixed prompt (~5.2k measured) plus that budget, so any model ≥ ~14 GB on a 16 GB card had ~2–3k tokens of real room and returned truncated: true on every step. The new value is derived rather than picked: minUsableContextWindow() names the requirement (fixed prompt + generation budget + boundary margin = 14704 on defaults) and a test asserts the floor clears it, so the two constants cannot drift into crossing again. Bootstrap now also warns when the reported contextWindow is under that figure, managed or external, naming the knob to raise — previously the budget code detected the impossibility, silently floored the conversation to 512, and said nothing.

                      before   after
gemma-4-26b-a4b        8192    16384
gemma-4-31b            8192    16384
qwen-3.6-27b           8192    16384
qwen-3.5-35b           8192    16384

5. Stop guessing the slot count (slot-manager.ts, model-profile-manager.ts, bootstrap.ts). SlotManager defaulted to 4 slots, and because managed mode always defers the boot health check that default outlived the probe — sessions were handed ids 1–3 against a --parallel 2 daemon. llama.cpp wraps out-of-range ids (id_slot % n_slots) rather than erroring, so nothing surfaced: the ids silently collided with another session's slot, evicting its KV cache and forcing a full ~5k-token prompt reprocess on every rotation.

• Default is now 1. Slot 0 is the only id every llama-server is guaranteed to have, so the pre-probe pool cannot overrun the server.
• SlotManager.resize() widens it once the real count is known. It drops stale assignments (a session's slot may no longer exist, and the server-side KV is not where we think either way — one honest cold rebuild beats pointing at a stranger's cache) and re-takes an out-of-range reflection reservation.
• ModelProfileManager already re-probes /props at turn start; it now reports total_slots through an onTotalSlots hook that bootstrap wires to resize(). Discovery runs before the profile/grammar work so a grammar rebuild failure cannot swallow it.

• extractTotalSlots moved from bootstrap.ts to model-profile.ts so the boot probe and the refresh path share one parser.

  default slot count : 1 (was 4)
  ids before /props : 0,0,0
  after resize(2) : 0,1,0 (matches --parallel 2)

Tests

• No catalog model ships a template override; any declared override must resolve on disk.
• ws is bounded for all three profiles.
• A timeout aborts once; genuine transport failures still retry.
• The context floor clears the agent requirement and stays under the auto-size ceiling.
• The default slot pool never emits an id above 0; resize widens, no-ops on an unchanged count, drops stale assignments, keeps in-range reservations, releases out-of-range ones, and hands the sole slot back when shrinking to one.
• onTotalSlots fires on a successful probe, stays silent when the field is absent, and still fires when profile detection finds nothing.

npm run lint and npm run build clean. Full suite: 3343 passed.

Pre-existing failures, untouched by this branch (confirmed by stashing these changes and re-running on a clean tree — identical set): tui-app, chat-log, splash-banner, llm-panel-selectors, persist-embedding-hybrid-recall, send-message-concurrency, fs-glob-real. Separately, tui/llm-health/llm-health-poller is a real-timer test (~450 ms/case) that flakes under full-suite load — it passed in isolation and on a re-run, and this branch does not touch it.

Upgrade note

--chat-template-file and --ctx-size are baked into the running daemon, so a rebuild alone changes nothing — models stop && models start is required. Verify with:

curl -s http://127.0.0.1:19091/props | grep -c think          # want > 0
grep n_ctx_slot ~/.atomic-agent/models/llama-server.log | tail -1   # want >= 16384

[Anonymous]

Ticket changed by: Ooooze

• status: open --> closed