#11 feat(intelligence): dynamic local LLM selection for task classifier (Apple Silicon)

[Anonymous]

Originally created by: Akarsh-Hegde

Summary

• Introduces select_model_for_hermes() in llm_selector.py — discovers the best available local OpenAI-compatible LLM endpoint and passes it to hermes AIAgent instead of always calling the static cloud config
• _ensure_mlx_server() manages a persistent mlx_lm.server process (PID file at ~/.meridian/mlx_lm_server.pid) so the model loads once and reuses across every run_task_linker.py subprocess invocation
• task_classifier_agent now calls select_model_for_hermes(budget_pct=LLM_BUDGET_PCT) before constructing AIAgent; falls back to static MODEL/BASE_URL if nothing local is available
• Fixes 19 pre-existing test failures from the classifier refactor (stale stage-flag tests removed, MCP summary regex broadened to match session(s) format)

How it works

Priority order on every classification call:

1. Reuse a running server — probes Ollama (11434), LM Studio (1234), llama.cpp/mlx_lm (8080); if any has a model loaded, hermes uses it immediately at zero load cost
2. Start mlx_lm.server — reads Metal GPU headroom via mx.device_info(), multiplies by LLM_BUDGET_PCT (default 0.5), selects the largest model that fits from the mlx-community catalog, starts mlx_lm.server on port 8765 as a persistent background process
3. Cloud fallback — if nothing is available (LLM_PREFER_LOCAL=0 or no headroom), hermes uses the existing OLLAMA_MODEL/OLLAMA_HOST config unchanged

Verified on M4 Max

headroom=28.1GB × 0.5 budget = 14.0GB → phi-4 (8.5GB) selected
task_classifier_agent: local model=mlx-community/phi-4-4bit runtime=mlx_managed
task_classifier_agent: model=mlx-community/phi-4-4bit base_url=http://127.0.0.1:8765/v1
→ task_key=KAN-105  confidence=0.85  routing=auto  elapsed=11.5s

Test plan

• [ ] 92/92 Python agent tests pass (cd services && python -m pytest agents/tests/ -v)
• [ ] 18 new test_llm_selector.py tests cover _select_mlx_entry, probe_compute, select_model_for_hermes, _ensure_mlx_server — all external I/O mocked, no Apple Silicon required
• [ ] Set LLM_PREFER_LOCAL=0 to verify cloud fallback path still works
• [ ] Set LLM_BUDGET_PCT=0.0 to verify nothing is selected when budget is zero
• [ ] With Ollama running (ollama serve): confirm runtime=ollama appears in logs and model loads in <1s

Config

Variable	Default	Purpose
LLM_PREFER_LOCAL	1	Try local model before cloud
LLM_BUDGET_PCT	0.5	Fraction of free GPU memory to allocate

Install local inference deps: pip install -e ".[local-llm]" (adds mlx-lm + psutil)

🤖 Generated with Claude Code

[Anonymous]

Ticket changed by: adityaharishch

• status: open --> closed

[Anonymous]

Originally posted by: adityaharishch

🎉 This PR is included in version 1.0.0 🎉

The release is available on:

• v1.0.0
• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀