#38 feat(evals): classifier experimentation harness + openobserve stability fixes

[Anonymous]

Originally created by: Akarsh-Hegde

Summary

This branch lands the classifier experimentation harness (KAN-139) — what started as a golden-dataset PR has grown into the full eval loop the team will use to evaluate every prompt edit, model swap, and config change going forward. Three eval runs are already in the bank, with their failures clustered into actionable prompt-rule candidates.

It also bundles a separate stream of OpenObserve daemon stability fixes that landed on this branch.

---

What's in here

1. Eval pipeline harness

• services/tests/evals/render_seeds.py — bridges hand-authored golden_seed/dev_<persona>_sessions.json (with structured ground_truth + design_notes) into the deepeval Golden shape that test_mlx_classifier.py consumes. Recent-context block uses only scoreable prior sessions, matching build_dataset.py:_fetch_recent so sub-scoreable timeline filler can't leak into the prompt.
• services/tests/evals/smoke_run.py — one-shot runner that emits OTel spans to OpenObserve under service.name=meridian-eval. One eval.run root + one eval.classify per Golden, with per-Golden force_flush so spans land live. Deepeval's @observe hijacks the global TracerProvider for Confident AI, so this OpenObserve-targeted runner is a deliberate separate process.
• services/tests/evals/test_mlx_classifier.py — reads dataset path from EVAL_DATASET_PATH env var, defaulting to .dataset.json. Same pytest suite now gates against either the legacy real-data export or the synthetic seed-rendered Goldens.

2. Golden dataset (KAN-139 deliverable)

• services/tests/evals/golden_seed/dev_a_sessions.json — Dev A persona expanded from 35 → 50 hand-authored sessions (40 scoreable + 10 sub-scoreable timeline filler). Coverage spans 7 difficulty tiers: easy, medium, hard, hard-decoy, overhead, untracked, context-only. Each scoreable Golden targets a documented failure mode (design_notes field) — chat-mention false positive, branch-name vs file-path disagreement, topical-adjacency, decoy resistance, etc.
• services/tests/evals/golden_seed/candidates_meridian.json — 5 real KAN tickets + 2 synthetic decoys (already on main).

3. Configurable session-text cap

• services/agents/_prompts.py — SESSION_TEXT_CAP is now env-overridable (SESSION_TEXT_CAP=0 disables truncation entirely). Default stays at 2500 chars for production safety. Used during the cap-experiment run to test whether truncation was the bottleneck (it wasn't — see Findings below).
• CLAUDE.md — documents the new env var.

4. Structured feedback log (FEEDBACK.json)

• services/skills/activity/task-classifier/FEEDBACK.json — accumulates per-run failures across eval runs as structured observations clustered into reusable failure_classes. Three top-level slices:
• runs[] — one entry per eval execution (trace_id, model, prompt_version, dataset, metrics, optional experiment block for A/B runs)
• observations[] — one entry per failing Golden per run, with classifier's full reasoning text and a failure_class_id foreign key
• failure_classes[] — synthesized actionable backlog with proposed_prompt_rule ready to copy into SKILL.md, plus status: open | resolved_in:<version> for tracking which prompts close which classes
• _meta.failure_class_schema — declared contract for required vs optional fields with defaults documented; aggregator scripts can do .get(key, default) safely.

5. Claude Code skill (.claude/skills/eval-feedback/)

• Self-contained skill invoked on demand after an eval run ("analyze the latest run", "what failed", etc.). It queries OpenObserve for the eval.run + eval.classify spans, reads FEEDBACK.json via jq (lean path — avoids loading the whole file into context as it grows), clusters failures, appends via a Python script (load → mutate → write), validates, and produces a ranked summary of open failure_classes for the next SKILL.md revision.
• Includes a documented gap: today the skill identifies the model via human-input or source-code default. A task #1 TODO documents the fix (add /info endpoint to services/agents/server.py + emit model_id as a span attribute on eval.run).

6. OpenObserve daemon stability (separate stream)

• Memory caps: ZO_MEMORY_CACHE_MAX_SIZE=2GB + ZO_DATAFUSION_POOL_SIZE=4GB — OO was auto-sizing to ~22.5 GB on a 36 GB machine.
• plist routes through run.sh: RUST_LOG and memory caps configurable without editing the plist; default log level warn (override: echo info > ~/.openobserve/.log_level).
• Evicted legacy ai.openobserve agent: was crash-looping 17,495 times (~23 GB of log waste).
• Install flow: prompts for email+password separately, computes base64 internally — new users were confused being asked for raw base64 before OO had started.

7. Docs

• TESTING.md §9 — full eval-pipeline section: input/output schema, run flow, failure-mode taxonomy, OpenObserve queries, when-to-re-run checklist.
• services/tests/evals/README.md — file inventory, run cookbook (smoke run / pytest / Confident AI flows), Golden schema, OpenObserve trace schema, OpenObserve-vs-Confident-AI decision matrix, known issues.
• CLAUDE.md — registers 5 new env vars (MERIDIAN_OTLP_ENDPOINT, MERIDIAN_OO_AUTH, MLX_SERVER_URL, EVAL_DATASET_PATH, SESSION_TEXT_CAP); "Add a Golden" common task; eval commands in the quick command reference.
• .gitignore — ignore .deepeval/ cache, .synthetic-dataset-*.json (regenerable artifact).

---

Runs logged so far (model: mlx-community/Qwen3.5-9B-OptiQ-4bit for all 3)

run_id	trace_id	Goldens	both	task_key	session_type	notes
phi4-4bit-dev_a-baseline-20260528	945c2f50…	26	80.8%	80.8%	84.6%	First end-to-end run. Baseline against 26-Golden subset.
smoke_20260528T162251	dfc053b2…	40	65.0%	67.5%	72.5%	Dataset expanded to 40 Goldens (same prompt). Surfaced 5 new failure_classes including the first false-negative class.
smoke_20260528T180202	3ae5eb31…	40	60.0%	65.0%	67.5%	Cap-experiment (SESSION_TEXT_CAP=0). Both-accuracy DROPPED vs control; finding documented below.

Model label correction (see commit 925b1e89): the first two runs were originally logged with model: mlx-community/phi-4-4bit based on conversation context. Verified against ~/.meridian/logs/mlx-server.log and the live launchd MLX process: the model has been Qwen3.5-9B-OptiQ-4bit since 2026-05-25. phi-4 is cached on disk but was never loaded by the running daemon. FEEDBACK.json model fields are corrected; run_id strings and the Confident AI run identifier (phi4-4bit-dev_a-baseline) carry the old phi-4 string as immutable history. The drift source is tracked in task [#1] (add /info endpoint).

---

Findings (in priority order)

Top open failure_class: optimism-bias (27 occurrences across 3 runs)

Cross-cutting parent of 6 specific classes. Every wrong answer in the false-positive direction was a task claim with a real ticket key at confidence ≥0.85. The classifier wants to find a ticket and given any thread to pull on, will pull. Proposed prompt rule: "Default to untracked when evidence is adjacent (mentions, related reading, topical overlap). Default to task when evidence is direct (active editing, terminal execution, debugging, or analysis whose output IS the ticket's deliverable). Cap confidence at 0.6 when only adjacent signals are present."

Cap experiment result: SESSION_TEXT_CAP is a covariate, not the bottleneck

Removing truncation (cap=0 vs cap=2500) made overall accuracy worse (65.0% → 60.0% both). But the per-tier movement was bidirectional:

• hard +15pp (71% → 86%) — more context helped disambiguation
• overhead −15pp, untracked −33pp, medium −12pp — more adjacent signals = more optimism-bias

Conclusion: prompt-design fix is the right next move; keep cap default at 2500.

Open failure_classes ranked

rank	class	occurrences	runs
1	optimism-bias	27	3
2	chat-mention-as-work	10	3
3	reading-as-doing	6	3
4	branch-name-locks-classification	4	2
5 (tied)	meta-file-edit-as-ticket-work, filepath-outweighs-branch, overhead-untracked-boundary, dataset-analysis-missed-as-ticket-work	3 each	2–3
9	pr-review-as-ticket-work	2	2
10	dogfooding-as-ticket-work	1	1

---

Follow-ups already tracked

• task [#1] — Add GET /info endpoint to services/agents/server.py returning {model_id, backend, max_tokens}; smoke_run.py queries at startup and sets model_id as a span attribute on eval.run; eval-feedback skill reads from that span instead of source-code defaults. Closes the model-label drift source.
• task [#12] — Cut SKILL.md v2.1 with the anti-optimism rule. Should close the top 5 failure_classes simultaneously (~50 of 62 total occurrences) based on the projected coverage. Re-run eval to verify, mark resolved classes resolved_in:2.1.0 in FEEDBACK.json.

---

Test plan

• [x] services/.venv/bin/python services/tests/evals/render_seeds.py produces .synthetic-dataset-a_meridian.json with 40 Goldens (50 sessions → 40 scoreable + 10 sub-scoreable)
• [x] smoke_run.py against MLX server on port 7823 lands 1 root + N child spans in OpenObserve (verified with per-Golden force_flush ensures live span delivery)
• [x] deepeval test run --identifier qwen35-9b-optiq-4bit-dev_a-baseline posts test-run + per-Golden traces to Confident AI
• [x] eval-feedback skill correctly queries OpenObserve, clusters failures, appends to FEEDBACK.json, validates JSON, and does not double-append previously-logged run_ids
• [x] _meta.failure_class_schema is the authoritative contract — verified no class uses an undeclared field
• [x] cargo fmt --check + cargo clippy -- -D warnings clean on every commit (pre-commit hooks)
• [ ] OpenObserve memory stays within caps after a 24h soak
• [ ] Cut SKILL.md v2.1 (task [#12]), re-run eval, verify which failure_classes close

🤖 Generated with Claude Code

[Anonymous]

Originally posted by: Akarsh-Hegde

PR description updated — what's new since the last version

Since the original PR body ("eval pipeline + OpenObserve stability"), 7 more commits landed that materially change what this PR ships. Highlights:

• FEEDBACK.json now exists as the structured backlog (3 runs logged, 35 observations clustered into 10 failure_classes) — the single most useful artifact for whoever picks up SKILL.md v2.1
• eval-feedback Claude Code skill automates the FEEDBACK.json maintenance loop end-to-end, with lean-read via jq and a Python load→mutate→write pattern that scales as the file grows
• _meta.failure_class_schema declares the required + optional field contract (9 required, 5 optional with documented defaults)
• Model-label correction — the first 2 runs were retroactively corrected from phi-4-4bit to Qwen3.5-9B-OptiQ-4bit after verification against the live launchd process and mlx-server.log. run_id strings preserved as immutable history; model fields fixed; correction note + drift source documented in FEEDBACK.json _meta
• SESSION_TEXT_CAP made env-overridable in services/agents/_prompts.py — default stays 2500; tested with cap=0 (no truncation) and result is logged in FEEDBACK.json with experiment metadata
• Findings (now in the body): optimism-bias is the cross-cutting top failure mode (27 occurrences across 3 runs); the cap experiment confirmed prompt-design is the bottleneck, not context size

Two follow-ups are tracked: task [#1] (/info endpoint to fix model-label drift authoritatively) and task [#12] (cut SKILL.md v2.1 with the anti-optimism rule).

[Anonymous]

Ticket changed by: adityaharishch

• status: open --> closed

[Anonymous]

Originally posted by: adityaharishch

🎉 This PR is included in version 1.0.0 🎉

The release is available on:

• v1.0.0
• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀