#327 feat(benchmarks): add labelled corpus metrics

[Anonymous]

Originally created by: tomatotomata

What this PR does

Implements the first complete slice of [#241]:

• adds a 100-row deterministic JSONL corpus under tests/corpus/benchmark.jsonl
• validates required fields, duplicate IDs, attack flags, and verdict thresholds before evaluation
• maps rows through the existing public benchmark action types and real decision engine
• adds redaction-safe per-category TPR, FPR, and precision metrics
• adds unit and integration gates for schema validation, verdict contracts, determinism, and payload redaction
• documents the corpus runner and metric definitions

Validation

• python -m pytest tests/unit/test_benchmark_agentdojo.py tests/unit/test_benchmark_corpus.py tests/unit/test_benchmark_harness.py tests/unit/test_benchmark_operator_model.py tests/integration/test_benchmark_corpus_gate.py tests/integration/test_benchmark_synthetic_gate.py
• 60 passed
• ruff check passed on changed Python files with the repository's existing Ruff 0.15 compatibility exclusions
• git diff --check passed

The full repository suite was also attempted, but the local run exceeded three minutes before producing a result.

[Anonymous]

Originally posted by: fu351

Saw the update land — thank you for taking [#241] on. Wiring the corpus rows through the real decision engine instead of a mock was the right call: it's what makes the TPR/FPR numbers worth trusting. The determinism and redaction gates are exactly the discipline this repo cares about, so you're building on the right foundations.

No rush from my side while it's in draft. Two things that might save you time:

• The branch currently conflicts with main — a few PRs merged this week. If the rebase turns into a slog, ask here and I'll point you at what moved.
• Don't worry about the full local suite timing out. Once the conflicts are resolved, CI runs the whole thing for you (pytest -n auto), so the targeted runs you listed are plenty locally.

Flip it to ready whenever you want eyes on it and I'll review promptly. Questions along the way are welcome here or on Discord. Great to have you building this.

[Anonymous]

Originally posted by: fu351

Closing this one, and you're owed a straight explanation, because closed should not read as rejected.

[#241] got implemented on main last week as [#370]: a 137-row corpus with per-category TPR/FPR/precision and a CI gate, building on [#322]'s harness-fidelity work. A security-audit deadline made the measurement gap urgent and I built it in-house rather than waiting on a draft that was sitting conflicted. The timing cost you the landing, and that's on the maintainer side, not on your design.

Worth saying plainly: your draft called several things right before [#370] existed. Schema validation before evaluation, determinism and redaction as explicit test gates, and running rows through the real decision engine instead of a mock are all in the shipped version too. The main design difference is calibration: [#370] derives each attack row's expected floor from the live engine (raise-only, so the corpus is a regression fence), where your draft pinned fixed thresholds.

If you're up for another measurement piece, [#243] (friction telemetry, level-6) is cut from the same cloth: interventions-per-session as the instrument for the alert-fatigue thesis, which the new corpus made measurable (almost all mitigation today is AUTH, not BLOCK). [#234] (level-4) is a smaller calibration slice if you'd rather warm up. Comment on either and it's yours.

[Anonymous]

Ticket changed by: fu351

• status: open --> closed

[Anonymous]

Originally posted by: fu351

@tomatotomata Sorry this one wound up closed; the labelled-corpus groundwork was real, and I hope it wasn't your last swing here.

Quick community roll-call: are you in the Doberman Discord? If yes, drop your handle here (or DM me there) so I can match the GitHub side of your work to your Discord name. If not, you're warmly invited: https://discord.gg/Sfy5XGNqty