#370 test(benchmarks): labeled detection corpus + per-category FPR/TPR harness (C8, #241)

[Anonymous]

Originally created by: fu351

Slice

• Feature / Slice: C8 — labeled detection corpus + FPR/TPR harness (closes [#241])

What this PR does

The benchmark suite proved the harness was wired correctly against a narrow synthetic case, but detection quality — the false-positive rate that drives approval fatigue, and per-category TPR — went unmeasured. This adds a flat, hand-editable JSONL corpus and a per-category metrics layer that reuses the existing mapping/runner/metrics.

• tests/corpus/detection_corpus.jsonl — ~137 labeled rows across injection / exfiltration / secrets / destructive / encoded / benign.
• suites/corpus.py — a schema-validated CorpusRow loader, a CorpusAdapter (the aggregate ASR/FPR + redaction path), and a per-row driver (evaluate_corpus) that keeps the row→verdict link the per-row assertions need.
• metrics.corpus_metrics() — per-category + overall TPR / tpr_strict / FPR / precision, plus the payload-free ids of any assertion violation.
• run.py --corpus — prints the per-category report in any F6 mode.
• tests/corpus/_generate.py — a dev tool that (re)builds and calibrates the shipped floors against the live engine.

The two properties that make it honest

1. Raise-only, calibrated floors. Each attack row's expected_verdict_at_least is the verdict the engine actually reaches today (null = a documented gap). The generator refuses to lower a shipped floor; the CI gate fails on any attack that regresses below it. Each benign row's forbidden_verdict_at_least is the false-positive guard — a legitimate action must not be over-blocked.
2. Not tuned to wins. Pure natural-language injection scores TPR 0.0 and the corpus says so (it's a provenance/subjective concern, not a content-pattern one). Calibration also surfaced a real precision note: reading an .env.example template over-blocks (the secret-path regex matches .env.* fail-closed).

Measured (built-ins only, balanced mode)

Category	n	TPR	tpr_strict	FPR
secrets	7	1.00	0.43	—
destructive	10	0.80	0.60	—
encoded	79	0.82	0.00	—
exfiltration (balanced / strict)	8	0.375 / 1.00	0.00	—
injection	8	0.00 (documented gap)	0.00	—
benign	25	—	—	0.00
Overall	112 atk / 25 ben	0.74	0.08	0.00

Precision 1.00, FPR 0.00 — but tpr_strict 0.08 says almost all mitigation is a human-gated AUTH, not a hard BLOCK (the "AUTH is a leash, not a wall" caveat, now measured per category).

Tests added (run in CI)

• tests/unit/test_corpus_benchmark.py — metrics math on a known fixture, schema validation of every malformed shape, payload routing, "a fresh JSONL drives with no harness change", redaction.
• tests/integration/test_corpus_gate.py — the deterministic real-engine gate: no benign over-block, no attack below its floor, report is redaction-clean.
• 24 tests, all green locally; ruff + lint-imports (3/3) clean.

Security checklist

• [x] Fails closed on error / uncertainty (a task-token/row error fails toward more protection; malformed rows raise rather than silently skip)
• [x] No secret, full file, or unredacted prompt logged or committed — payloads ride only in raw_arguments, never a report; the secrets category triggers on credential paths + shapeless high-entropy values, never assembled provider literals (push-protection-safe)
• [x] Any guardrail/learning change is raise-only — floors are raise-only; no engine behavior changes (tests-only)
• [x] Every BLOCK/AUTH carries reason codes + a human explanation — unchanged

Public-release safety

• [x] Nothing enterprise/proprietary — benchmark tooling + synthetic fixtures only

Repo

• doberman-core (public)

[Anonymous]

Ticket changed by: fu351

• status: open --> closed