Menu

#235 Optional [perplexity] extra: reference windowed scorer for the statistical OOD channel (OT.4)

open
nobody
2026-08-06
2026-08-03
Anonymous
No

Originally created by: fu351

The deterministic Unicode scanner cannot see a GCG-style adversarial suffix: the red-team corpus proves those fixtures pass it clean, and the statistical channel that would catch them has stayed inert because core ships no scorer. Ship the reference scorer as an optional extra.

Build

  • src/doberman/perplexity.py: build_perplexity_fn(model_name="gpt2", window, stride) with lazy transformers/torch imports, scoring the max over sliding token windows so a local suffix inside long benign text still scores high; maybe_build_detector() returns a TokenChannelDetector(perplexity_fn=..., perplexity_threshold=...) or None.
  • pyproject.toml: a new perplexity extra (transformers, torch). Never a default or dev dependency; default CI runs no model.
  • Wiring: the proxy's default subjective stack appends the detector via extra_detectors. The host-hook path never imports this module (tested via a clean-interpreter sys.modules check).

Double opt-in: the extra must be installed AND DOBERMAN_PERPLEXITY=1 set. Without either, behavior is byte-for-byte unchanged.

Guarantees (each one is a test)

  • Raise-only: soft channel, AUTH-capped by the subjective layer; it can never lower a verdict or hard-block.
  • Model-load failure or a raising scorer means the statistical channel is simply absent; every deterministic protection is untouched.
  • Scores only; scored text is never logged or persisted.
  • The shipped default threshold is conservative; operators recalibrate with calibrate_perplexity_threshold on their own benign corpus.

Honest scope: catches high-perplexity adversarial text within a calibrated FPR budget. Defense-in-depth, not a jailbreak detector.

Blocked by [#234] (the calibration helper lands first).

Related

Tickets: #234
Tickets: #445

Discussion


Log in to post a comment.