#310 feat(classifier): TASK GATE prompt + DeepEval untracked-guard eval (70%→96% guard)

[Anonymous]

Originally created by: adityaharishch

Summary

Rewrites the session task-classifier prompt and adds an end-to-end DeepEval suite that measures the failure mode that matters most: forcing non-task ("untracked") work onto a ticket. Net result on 103 real hand-labeled sessions: untracked-guard 70% → 96%, zero real tasks lost (task recall 100%).

What's in here

Prompt (SKILL.md)

• Adds a hard TASK GATE — assign a task_key only when (1) hands-on production not viewing, (2) scope match not topic/app/repo match, (3) the ticket is a listed candidate.
• Adds a "NOT evidence of working a ticket" list (key merely visible on screen; viewing/monitoring dashboards/traces/DB/PRs/logs; different repo/product; recency alone).

Production fixes

• fix(classifier): clamp confidence/category_confidence to [0,1]. Outlines' FSM constrains JSON structure/type but not numeric range, so a model-emitted -0.85 was crashing model_validate_json and silently dropping the entire verdict.
• perf(mlx-server): the /classify endpoint now uses the same inference core as production classify_session (cached FSM logits-processor), cutting latency ~34s → ~20s/call. (Prefix KV-cache stays disabled on this model — its cache is non-trimmable — so ~20s is the floor for the full session_summary.)

Eval (services/tests/evals/)

• DeepEval pytest suite (test_mlx_e2e) over 103 real goldens with deterministic BaseMetrics: UntrackedNotTask (the guard) + TaskKeyMatch + SessionTypeMatch.
• 103 hand-labeled real sessions across 3 sets: tuning (49), held-out validation (28), false-positive stress (26).
• build_real_goldens.py rebuilds the byte-exact production prompt per session.

Results (DeepEval, 103 goldens)

Metric	Pass rate
UntrackedNotTask (guard)	96.1%
TaskKeyMatch	87.4%
SessionTypeMatch	81.6%
Task recall	100% (10/10)

Validated on 54 unseen sessions (held-out guard 100% / 92%) → generalizes, not overfit.

Notes

• data/generated/ (built goldens) is gitignored — rebuilt from the labels via build_real_goldens.py.
• skills-lock.json (deepeval skill hash) was intentionally left out — unrelated to this change.

[Anonymous]

Originally posted by: Akarsh-Hegde

Bug: routing removed from Python response — untracked work queue will be permanently empty in the dashboard

_classify_one return dict (lines 1567–1579) no longer includes a routing key. Rust's SessionClassification struct has #[serde(default = "default_routing")] returning "pending" (mod.rs:132), so db_write.rs:82 writes task_routing = 'pending' for every newly classified session.

The dashboard buckets untracked work exclusively on routing === 'queue':

• ui/components/views/TodayView.tsx:491: else if (s.routing === 'queue') pushToBucket('_queue', bs) — never fires for new sessions
• TodayView.tsx:553: queueCount = .filter(s => s.routing === 'queue').length — always 0
• ui/components/views/SessionsView.tsx:102 and TaskBadge.tsx:52,91 also key on routing values

Note: collect.rs already filters on task_session_type = 'task', not task_routing, so worklog generation is unaffected. The breakage is dashboard-only but high-impact: the untracked session queue — where real work that didn't match a ticket surfaces for review and ticket creation — will appear empty for all sessions classified after this PR.

Fix options (pick one):

1. Add "routing": "auto" if task_key else ("queue" if result.session_type == "untracked" else "skip") back to the return dict in _classify_one.
2. Update TodayView.tsx, TaskBadge.tsx, SessionsView.tsx to bucket on session_type instead of routing (longer-term cleaner).

The Rust struct comment says "Routing is computed by Rust, not the server" suggesting option 2 is the intended direction — but the Rust code hasn't been updated to implement it yet.

[Anonymous]

Originally posted by: Akarsh-Hegde

Bug: category_explanation removed from Python response — worklog comment bullets silently dropped for all new sessions

The Python response no longer includes category_explanation. today/route.ts in this PR already patches the dashboard tooltip by falling back to task_reasoning — good. But the PM worklog writer (src/pm_worklog/route.rs:232) was not updated and still reads category_explanation directly:

// src/pm_worklog/route.rs:232
if let Some(expl) = s.category_explanation.as_deref().filter(|t| !t.is_empty()) {
    // builds "category: ..." bullet in the Jira/Linear comment
}

Since db_write.rs converts the empty serde default ("") to NULL in the DB, this condition is always None for new sessions — the category justification bullet silently disappears from every Jira/Linear/GitHub worklog comment generated after this PR, with no error or warning.

Fix: Either update src/pm_worklog/route.rs to fall back to task_reasoning the same way today/route.ts does, or build the bullet from the classification result's reasoning field (which is now always present and already stored in the task_reasoning column).

Also: the Rust unit test at src/intelligence/task_linker/mod.rs:688-703 (deserializes_real_mlx_server_response_with_category_fields) hardcodes category_explanation in the fixture JSON and asserts its value. The test still passes (static fixture), but it no longer reflects the live server wire format — the production path always gives category_explanation: "". Consider adding a test with the current wire shape to guard the fallback path.

[Anonymous]

Originally posted by: Akarsh-Hegde

Bug: Apple FM hallucinated task_key discards real work as overhead instead of retaining it as untracked

_coerce_apple_fm_result() does not validate task_key against the candidate set. When Apple FM (not FSM-constrained) emits session_type: "task" with a plausible-but-fabricated key, the key passes through coercion unchanged and reaches _classify_one:1516:

if task_key is not None and task_key not in valid_keys:
    return _error_result(session_id, f"model returned unknown task_key {task_key!r}", ...)

_error_result returns session_type="overhead" with confidence=0.0 — a hard discard. Real work the model recognised (but assigned to the wrong key) is thrown away instead of being retained as untracked.

The FSM path cannot produce this failure because it is structurally constrained to the candidate set. The Apple FM path is the only code path that can generate a semantically invalid task_key, and it 's the one without the guard.

Fix: Pass valid_keys into _coerce_apple_fm_result and null out an out-of-set key, downgrading to untracked:

def _coerce_apple_fm_result(data: dict, valid_keys: frozenset[str]) -> dict:
    ...
    if data.get("session_type") == "task" and data.get("task_key") not in valid_keys:
        data["task_key"] = None
        data["session_type"] = "untracked"
        data.setdefault("confidence", 0.65)

[Anonymous]

Originally posted by: adityaharishch

Thanks @Akarsh-Hegde — went through all three. Verdict: #3 is a real bug and is now fixed; [#1] and [#2] are not regressions from this PR. Details below.

---

✅ [#3] — Apple FM hallucinated task_key discarded as overhead — FIXED (20af4be)

Correct and real. The Apple FM path (model_id == APPLE_INTELLIGENCE_ID) is not FSM-constrained, so it can emit session_type:"task" with a fabricated key, which the valid_keys guard in _classify_one then hard-discards as overhead (confidence 0.0).

Implemented your suggested fix: _coerce_apple_fm_result now takes valid_keys and nulls an out-of-set key, downgrading the verdict to untracked so the recognised work is retained. Threaded valid_keys through _classify_apple_fm → _parse → coercion. The FSM path is structurally immune, so no change there.

---

❌ [#1] — routing removed → empty untracked queue — not a regression (declining)

routing was never a key in the Python response — git log -S'"routing"' -- services/agents/run_task_linker_mlx.py returns nothing across all history, and it's absent from main's return dict too. Rust's default_routing() → "pending" has therefore always been the value written for MLX-classified sessions; this PR changed nothing here.

Untracked work is also not lost: TodayView.tsx's final else branch buckets task_key==null && routing!='queue' && session_type!='overhead' sessions into _untracked (line 491). The _queue bucket (routing==='queue') is a separate, explicitly-flagged-for-review lane that the MLX path has never populated — a pre-existing TODO ("routing computed by Rust" was never wired), out of scope for a prompt-rewrite PR. Happy to track it as its own issue if you'd like the Rust-side routing logic implemented.

---

❌ [#2] — category_explanation removed → worklog comment bullets dropped — misattributed impact (declining)

The field removal is real, but src/pm_worklog/route.rs:232 is a lineage trace span (tracing::info_span!), not worklog comment text — it only affects the OpenObserve trace tree. And even there the loss is negligible: the category name is still emitted on the classification_verdict span (line 224), and the richer task_reasoning is emitted on the reasoning span (line 228), which is always present now.

The actual Jira/Linear/GitHub comment text is generated by the Python synth, which doesn't read category_explanation at all (no references in services/agents/pm_worklog_update/). today/route.ts already falls back category_explanation || task_reasoning for the dashboard tooltip. So no comment bullet is dropped. The removal was intentional — task_reasoning is the justification carrier going forward.

Good catch on the stale unit test fixture though — mod.rs:688 still hardcodes category_explanation; it passes (static fixture) but no longer mirrors the wire format. Noting it; not blocking this PR.

[Anonymous]

Ticket changed by: adityaharishch

• status: open --> closed

[Anonymous]

Originally posted by: adityaharishch

🎉 This PR is included in version 1.60.0 🎉

The release is available on:

• GitHub release
• v1.60.0
• npm package (@latest dist-tag)

Your semantic-release bot 📦🚀