#325 test(benchmarks): subjective-layer × AgentDojo baseline-separation diagnostic

[Anonymous]

Originally created by: fu351

What this PR does

Adds a diagnostic that exercises Doberman's adaptive subjective layer against the AgentDojo benchmark — the harness previously measured only the objective/deterministic floor. Measurement-only: no src/ change.

It warms a per-suite streaming baseline on AgentDojo benign traces through the production observe() path (this replay is the cold seed the subjective layer trains on), scores held-out-benign + injection traces through the production surprise_blended(), and reports a Mann-Whitney AUC distribution-separation diagnostic (benign vs attack surprise), per suite and pooled.

The leakage firewall (the crux)

Algebra.provenance derives 1:1 from SecurityObject.source_context, and the AgentDojo adapter sets source_context by ground-truth label (attack→tool_output, benign→user). A provenance-driven separation would therefore measure the adapter's labels, not the guard. So the eval runs two arms:

• provenance_free (the honest number): source_context is neutralized to a constant before inference, which closes both leak channels — the provenance enum and the confidence scalar (0.8 vs 0.7) that flows into the HST/novelty terms. The remaining separation rides only structural features (capability/destination/blast-radius derived from the real call), i.e. the genuine attack signature.
• with_provenance: reported only to quantify how much the label leaks. Never the headline.

Honesty guarantees

• Not an ASR, never threshold-tuned; AgentDojo is never a target metric.
• Allowed-only learning — warms on benign (allowed) traces only; attack + held-out actions are scored, never learned (enforced by a total_observations == warm_count assertion, since observe() swallows its own failures).
• Warm-sufficiency caveat surfaced per suite (cold_start_active, hst_engaged, n_warm_observations) — AgentDojo suites are small, so the HST abstains and a suite riding the constant prior is reported inconclusive.
• Redaction — scores/counts/suite names only, never a payload.

Tests added (run in CI)

• tests/unit/test_benchmark_subjective.py (7): the firewall proof (two actions differing only in source_context → byte-identical algebra in the honest arm, divergent in the leak arm), allowed-only, separation sanity, determinism, redaction, grouping/holdout.
• tests/unit/test_benchmark_agentdojo_live.py: a skipif-guarded live smoke (skips without the operator-supplied agentdojo package; runs no model).

How this was built & reviewed

Planned by a Fable 5 advisor (scoped to measurement-only — a shippable AgentDojo-derived seed was rejected as trained-on-test and deferred as a separate generic seeder); executed by the session model + Sonnet; binding pre-commit review by a separate fresh Fable 5 reviewer → APPROVE (adversarial on the label-leakage question; confirmed source_context is the only label-conditioned field). Three non-blocking review nits folded in (bucket-asymmetry disclosure + two clarifying comments).

Verification

ruff check . / ruff format --check . clean; lint-imports 2/2 kept; subjective tests 7/7; benchmark/touched-area suite 59 pass; live smoke skips. Reproduce: python -m tests.benchmarks.run --suite agentdojo --subjective.

Follow-up (open decision)

Whether to also build a generic doberman baseline seed --from <operator's-own-benign-traces> cold-start seeder (never AgentDojo-derived) — deferred pending a maintainer call.

[Anonymous]

Ticket changed by: fu351

• status: open --> closed