#47 Managed llama.cpp never returns a response: bundled Qwen 3.5 chat template disables reasoning and deadlocks the tool-call grammar

[Anonymous]

Originally created by: sachin-detrax

Summary

In managed local-LLM mode, submitting a prompt never returns a response. The GPU and CPU stay pegged and no tokens are ever streamed to the UI. The same model on the same hardware works fine in external mode (running llama-server myself and pointing localModels.url at it), which is what isolates this to the managed daemon's launch flags.

qwen-3.5-4b is DEFAULT_LLAMACPP_MODEL_ID, so this hits every managed-mode user on the default model.

Environment

• GPU: RTX 5070 Ti (16 GB)
• CPU: Ryzen 9 9900X
• RAM: 32 GB
• Mode: localModels.mode = "managed"
• Models tried: a large model, and qwen-3.5-4b (auto-sized to 32k context — so this is not a context-budget problem)

Steps to reproduce

1. localModels.mode = "managed", model qwen-3.5-4b.
2. atomic-agent models start.
3. Submit any prompt.
4. GPU/CPU load rises, generation clearly starts, no token ever appears, the turn never completes.

Root cause

models-catalog.ts attaches a bundled chat template to qwen-3.5-4b and qwen-3.5-35b:

chatTemplateAsset: "qwen3.5-chat-template.jinja",

startDaemon forwards it as --chat-template-file. That file is a Qwen2.5-style ChatML template with no thinking markers at all:

$ grep -c think assets/ai-models/qwen3.5-chat-template.jinja
0

llama-server echoes the override back at /props.chat_template, and that field is the only signal detectModelProfile has. looksLikeQwenThinkModel requires the template to contain <think> plus enable_thinking/preserve_thinking, so detection falls through to plain-instruct.

A/B with the same model and alias, only the template differing:

MANAGED (--chat-template-file bundled)
  profile : plain-instruct
  root    : root ::= tool-call-array
  prompt  : no reasoning prefill
  1st char the sampler may emit: "[" ONLY

EXTERNAL (GGUF's own template)
  profile : qwen-think
  root    : root ::= think-prelude tool-call-array
  prompt  : ends with <think> prefill
  1st char the sampler may emit: any (reasoning prelude)

buildGrammar early-returns for a reasoningStyle: "none" profile, so the reasoning prelude is stripped out of the GBNF root entirely.

Why that hangs

A reasoning model is now forbidden from reasoning. Its highest-probability first token is <think>; the grammar masks everything except [. It emits the forced [ and lands immediately in:

tool-call-array ::= "[" ws tool-call ( ws "," ws tool-call ){0,15} ws "]"
ws ::= [ \t\n\r]*

ws is unbounded, so whitespace is always legal — and llama.cpp hard-masks EOG to -INFINITY until a grammar stack empties (src/llama-grammar.cpp), which cannot happen until the whole array closes. The model has an infinite legal move and takes it.

This is the same failure class already documented at the reasoning seam in build-grammar.ts ("the sampler keeps emitting newlines until max_tokens instead of converging on the ["), which was bounded there with prelude-trail-ws — but ws itself stayed unbounded at ~10 seams inside the JSON body.

Why it is completely silent

assistant_delta is only emitted once the stream parser reaches {"tool":"reply","args":{"text":". Having consumed the [, the parser sits in its json_tool state matching /"tool"\s*:\s*"([^"\\]+)"/ against whitespace forever:

StepEvents emitted after '[' + 4000 newlines: 0

No assistant_delta, no reasoning_delta. Decode keeps the GPU busy, grammar sampling keeps the CPU busy, and nothing renders.

Why it looks infinite rather than erroring after 5 minutes

isRetryableLlamaError treats status === null as a transient transport failure. The client's own requestTimeoutMs (300 s) abort produces exactly that shape — the if (signal?.aborted) throw err guard only checks the caller's signal, not the internal timeout controller. So a merely-slow request is re-issued up to completionRetries (3) times: ~15 minutes of silent GPU churn before anything surfaces. The repair path uses the non-streaming complete(), which sits fully inside the retry wrapper.

Two related defects found while tracing

Neither triggers the hang, but both make it worse and both misbehave independently.

1. MIN_AUTO_CONTEXT (8192) equals the default completionMaxTokens (8192). The agent's fixed prompt measures ~5.2k tokens on its own, so a model landing on the floor has ~2–3k tokens of real room — not enough to close a reasoning block plus a tool-call array. Every step hits llama.cpp's context ceiling and returns truncated: true. Any model ≥ ~14 GB on a 16 GB card lands on this floor:

gemma-4-26b-a4b   14.25GB -> 8192
gemma-4-31b       17.29GB -> 8192
qwen-3.6-27b       17.6GB -> 8192
qwen-3.5-35b         22GB -> 8192

computeEffectiveConversationCap already detects the impossibility (its available goes deeply negative and floors the conversation to 512) and reports nothing.

2. SlotManager defaults to 4 slots against a --parallel 2 daemon. Managed mode always defers the boot health check (the daemon may not be up yet), so resolveModelProfile short-circuits and returns totalSlots: null — the default outlives the probe. ModelProfileManager refreshes profile and grammar at turn start but never the slot count. llama.cpp wraps out-of-range ids (id_slot % n_slots) rather than erroring, so nothing surfaces: the ids silently collide with another session's slot, evicting its KV cache and forcing a full ~5k-token prompt reprocess on every rotation.

Expected

Managed mode should behave like external mode for the same model: reasoning profile detected, tokens streamed, turn completes.

Workaround

Use external mode, or switch to qwen-3.5-9b — it has no chatTemplateAsset, so managed mode already detects qwen-think correctly.

[Anonymous]

Ticket changed by: Ooooze

• status: open --> closed