#112 feat(agents): dynamically select the MLX server model via llm_selector

[Anonymous]

Originally created by: Akarsh-Hegde

What

The MLX server (server.py --backend mlx) loaded one hardcoded model (Qwen3.5-9B-OptiQ-4bit) at startup via run_task_linker_mlx._MLX_MODEL_ID and served it to every stage — /classify_sessions, /summarise, /v1/chat/completions, /synthesise_worklog. This wires the existing llm_selector into that single load point so the model is chosen at runtime from available Metal headroom, while keeping the eval-tuned default on machines that can run it.

How

• llm_selector.py — select_mlx_model_id(): a new MLX-in-process selector. Deliberately does not reuse select_model_for_hermes/local_infer (which prefer external Ollama/LM Studio/Apple FM servers with no FSM-constrained decoding, and spawn a separate managed mlx_lm.server that would collide on the port). Priority: tuned model when it fits the budget → largest catalog model that both fits and is already in the HF cache → tuned model best-effort.
• _hf_model_cached(): gates catalog picks to weights already on disk, so dynamic selection never triggers a surprise multi-GB download (online) or an offline startup failure inside mlx_lm.load — on exactly the constrained machines this degradation path targets.
• Catalog entry: added Qwen3.5-9B-OptiQ-4bit to _MODELS at its 6.5 GB size slot (between gemma3-12b and qwen3.5-4b), so resolve_model() and the Hermes oscillation guard resolve it.
• run_task_linker_mlx.py: _MLX_MODEL_ID is resolved lazily/cached by _resolve_model_id() — explicit MLX_MODEL_ID pin > dynamic selection > hardcoded default (never returns None). Catalog is the single source of truth for the preferred model's footprint.
• server.py: /info and /v1/models report the resolved id, so the eval pipeline (which reads /info) stays truthful.

Why prefer-then-degrade

OptiQ is the eval-tuned classifier; the catalog's generic quality_score would otherwise silently swap it for llama3.3-70b on a big box. So capable Macs keep OptiQ; only RAM-constrained Macs degrade — and only to a model already present.

Testing

Verified end-to-end on an M4 Max (production server on :7823 untouched, test server on :7824):

• Dynamic: selector logged preferred fits budget=14.0 GB → loaded OptiQ; /classify_sessions (real session → KAN-141, coding, mlx_direct) and /summarise (clean FSM summary, no CoT leak) round-tripped; /info + /v1/models reported the selected model.
• Pin: MLX_MODEL_ID=Qwen3.5-4B bypassed selection; /info reflected the pin.
• Cache-gated degradation (forced headroom): 5 GB → Qwen3.5-4B (cached); 4 GB → falls back to OptiQ (only budget-fitting catalog model was uncached) — no download; thermal-throttle cap honored.

MLX_MODEL_ID env still pins a specific model for eval reproducibility.

🤖 Generated with Claude Code

[Anonymous]

Ticket changed by: adityaharishch

• status: open --> closed

[Anonymous]

Originally posted by: adityaharishch

🎉 This PR is included in version 1.9.0 🎉

The release is available on:

• v1.9.0
• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀