#234 calibrate_perplexity_threshold: the model-agnostic half of the perplexity seam (OT.3)

[Anonymous]

Originally created by: fu351
Originally owned by: AmirF194

TokenChannelDetector shipped with an opt-in perplexity_fn seam (engine/detectors/token_channels.py), but there is no principled way to pick perplexity_threshold; operators would have to guess. Add the calibration half the seam was designed around:

calibrate_perplexity_threshold(benign_scores: Sequence[float], target_fpr: float) -> float

in src/doberman/tokens.py. Pure stdlib. No model, no new dependency. It returns the empirical (1 - target_fpr) quantile of the benign scores, so any scorer can be calibrated against a benign corpus at a fixed false-positive budget.

Fail closed on bad input

• target_fpr outside (0, 1) raises ValueError
• fewer than 20 samples raises ValueError (too little data to promise an FPR)

Tests must cover: the returned threshold actually meets the target FPR on a synthetic score set; unsorted input; both ValueError paths; saturated scores (all 1.0). Then extend tests/unit/test_detector_token_channels.py with one wiring test: a stub perplexity_fn plus a calibrated threshold escalates a known adversarial fixture from tests/redteam/fixtures/ (AUTH, anomalous_token_pattern) and passes a benign string.

Document the seam contract in the docstring: perplexity_fn(text) -> float in [0, 1], higher = more anomalous, score >= threshold escalates. Core ships no model; this is the pure-stats half, and it unblocks the optional model-backed scorer.

[Anonymous]

Originally posted by: AmirF194

Picking this up. I will add calibrate_perplexity_threshold to src/doberman/tokens.py (nearest-rank empirical quantile, fail-closed on target_fpr outside (0, 1) and on fewer than 20 samples), plus the unit tests and the token_channels wiring test.

[Anonymous]

Originally posted by: fu351

Welcome, @AmirF194, and assigned. The plan reads right: nearest-rank quantile means the threshold is always a value that actually occurred, and refusing to calibrate on fewer than 20 samples or a target_fpr outside (0, 1) is the fail-closed shape everything in this repo follows. One thing for the tests: build the score arrays by hand or from a seeded generator; the suite has to be deterministic, so unseeded randomness won't fly. Ask anything on this thread as you go. And if you'd like to talk shop while you build, the pack hangs out on Discord: https://discord.gg/Sfy5XGNqty

[Anonymous]

• assigned_to: AmirF194

[Anonymous]

Ticket changed by: fu351

• status: open --> closed