#445 feat(tokens): add calibrate_perplexity_threshold for the OT.4 perplexity seam

[Anonymous]

Originally created by: AmirF194

Pull Request

Slice

• Repo: doberman-core
• Feature / Slice: [#234] (calibrate_perplexity_threshold, the model-agnostic half of the OT.4 perplexity seam)
• Plan reference: n/a (self-contained level-4 issue)

What this PR does

TokenChannelDetector (src/doberman/engine/detectors/token_channels.py) accepts an opt-in
perplexity_fn seam for the statistical OOD token channel, but nothing in the tree could pick a
perplexity_threshold from real data. Issue [#235] (the reference windowed scorer) names this as
its blocker: "Blocked by [#234] (the calibration helper lands first)".

Adds doberman.tokens.calibrate_perplexity_threshold(benign_scores, target_fpr), returning the
nearest-rank empirical (1 - target_fpr) quantile of a benign score corpus: the smallest scored
sample at or above that rank. Nearest-rank rather than interpolated, so the returned threshold is
a value that actually occurred in the corpus, at the cost of granularity no finer than
1 / len(benign_scores): the measured false-positive rate can exceed target_fpr by up to one
sample (n=20, target 0.10 measures 0.15 in the worst case). Fails closed: raises ValueError on a
target_fpr outside (0, 1), fewer than 20 benign scores, or any non-finite score, since none of
those can honestly promise the requested false-positive rate.

Tests added (run in CI)

• test_calibrate_perplexity_threshold_meets_target_fpr, ..._ignores_input_order,
  ..._rejects_bad_target_fpr, ..._rejects_too_few_samples, ..._handles_saturated_scores,
  ..._rejects_non_finite_scores
  (tests/unit/test_tokens_scanner.py), all with fixed, seed-free score arrays per your note on
  the issue.
• test_calibrated_threshold_escalates_gcg_suffix_and_passes_benign
  (tests/unit/test_detector_token_channels.py): wires a calibrated threshold through
  TokenChannelDetector with a stub scorer against a real GCG-style adversarial suffix from
  tests/redteam/fixtures/gcg_suffixes.txt (AUTH) and an ordinary benign string (PASS).
• pytest -n auto --cov=doberman --cov-report=term-missing --cov-fail-under=80: 90.7% total,
  tokens.py 96%, no line this PR adds is in the missing set.
• ruff check ., ruff format --check ., lint-imports,
  python -m tools.parity.generate_parity --check: all clean.

Not checked: the four real-Tk tests in test_gui_prompter.py fail in my Docker verification
image (libtk8.6.so missing from python:3.13-slim), identically on a clean main checkout in
the same image; unrelated to this change and not introduced by it.

Public-release safety (doberman-core only)

• [x] Contains nothing from the "not allowed" list: no enterprise/hosted code, no proprietary detection, no customer data, no secrets, no commercial-license code
• [x] Core still builds/tests/runs with NO enterprise package installed

Security checklist

• [x] Fails closed on error / uncertainty: bad target_fpr or too little data raises rather than returning a threshold that looks calibrated but is not
• [x] No secret, full file, or unredacted prompt logged or committed
• [x] Any guardrail/learning change is raise-only (no silent loosening): this is a pure stats helper with no side effects; it does not wire into any detector or change default behavior on its own
• [x] Every BLOCK/AUTH carries reason codes + a human explanation: unchanged, this PR adds no new call site into the detector
• [x] doberman-core does not import doberman_enterprise

Edge cases covered / Deviations from plan / Risks introduced

• Nearest-rank was chosen over interpolation per your comment on the issue: the returned
  threshold is always a value that actually occurred, at the cost of granularity no finer than
  1 / len(benign_scores).
• No wiring into TokenChannelDetector or a default scorer here; [#235] (the reference windowed
  scorer) is the natural next step and stays a separate PR.

---

Written with AI assistance (Claude Code); every command in the verification section above was
run this session before pushing.

Fixes [#234]

[Anonymous]

Originally posted by: fu351

Thanks @AmirF194, this is a strong first PR. The boundary tests hit the exact 19/20 and (0,1) edges, and reusing the real GCG fixture for the wiring test proves the seam against the actual attack class instead of a toy string. Two things before it merges:

1. A NaN in benign_scores sails through sorted() and can become the returned threshold, and downstream every score >= nan comparison is False, which silently disables the perplexity channel. Add a finiteness check that raises ValueError on any non-finite sample, same fail-closed shape as your other two guards, plus a test with a NaN in the corpus.
2. The PR text promises "at most target_fpr scores at or above the threshold", but nearest-rank overshoots that by up to 1/n (your N=20, fpr=0.10 case measures 0.15). The code matches the issue spec, so keep the math and fix the wording to say the measured FPR can exceed the target by up to one sample.

The red 3.11 leg wasn't yours, river 0.26.0 broke that leg repo-wide and the pin is merged as [#452]. The merge conflict it created on your branch is also mine to deal with, I'll land it from here once your two changes are in.

[Anonymous]

Originally posted by: AmirF194

Both addressed and pushed (767d436).

1. Added a finiteness check right after the sort, same fail-closed shape as the other two: any NaN or inf in benign_scores now raises ValueError instead of silently disabling the channel. New test with both a NaN and an inf sample.
2. Reworded the docstring and PR body to state the measured overshoot (up to one sample, 0.15 in your 20/0.10 example) instead of promising "at most target_fpr".

Left the merge conflict from [#452] to you as you said. Ran the tokens test file plus ruff/format/lint-imports clean in Docker; the full suite is timing out in my sandbox for reasons unrelated to this change, so I have not re-confirmed the full coverage run this round.

[Anonymous]

Ticket changed by: fu351

• status: open --> closed

[Anonymous]

Originally posted by: fu351

Merged, thanks @AmirF194! The follow-up commit that rejects non-finite scores is the part I like most, a NaN would have made every score >= threshold comparison false and quietly switched the perplexity channel off, and you caught that before I did. I broke the sort in calibrate_perplexity_threshold locally and test_calibrate_perplexity_threshold_ignores_input_order went red straight away, which is the kind of test I want here. Welcome to Doberman! If you'd like the next piece of this seam, [#235] (level-6) is the windowed scorer that consumes the threshold you just calibrated, and [#143] (level-4) is a similar wire-an-interface job if you'd rather something smaller first. Feel free to join the Discord if you have questions: https://discord.gg/Sfy5XGNqty

[Anonymous]

Originally posted by: AmirF194

Thanks for the review and the merge. The NaN edge case was the sharper catch, that channel would have gone dark with no error. Appreciate the pointers to [#235] and [#143], I'll take a look.