#241 Labeled detection corpus + FPR/TPR benchmark harness

[Anonymous]

Originally created by: fu351

The benchmark suite proves harness correctness (3 attack + 3 benign synthetic cases) but there is no labeled corpus measuring detection quality — so false-positive rate, the thing that drives approval fatigue, is unmeasured.

Build

• A JSONL corpus under tests/corpus/ with fields id, kind, surfaces, payload, is_attack, expected_verdict_at_least, forbidden_verdict_at_least, notes. forbidden_verdict_at_least is the false-positive guard: a benign row asserting it must never reach AUTH.
• Extend the benchmark runner with aggregate FPR/TPR/precision per category; the deterministic tier runs in CI.
• Seed ~100–150 rows across injection / exfiltration / secrets / benign, drawing on the red-team technique generator plus hand-labeled benign traffic.

Tests must prove: metrics math on a fixture corpus; schema validation of every row; CI runs the deterministic tier green; adding a row requires no harness change.

Adding labeled rows afterward is beginner-friendly standing work — this issue builds the harness that makes those contributions possible.

[Anonymous]

Originally posted by: tomatotomata

I’m taking [#241]. I was thinking of starting with the deterministic corpus schema and benchmark aggregation, then wiring the existing CI tier to it without changing detector behavior. I’ll keep the first pass self-contained and make each corpus row stand on its own so later additions do not need harness edits.

[Anonymous]

Ticket changed by: fu351

• status: open --> closed