#197 engine: add a PyPI API-token pattern to the secret-detection rule

[Anonymous]

Originally created by: fu351
Originally owned by: Varshith-Kali

What

src/doberman/engine/rules/secrets.py's _CREDENTIAL_PATTERNS tuple (lines 52–83) already recognizes AWS, OpenAI, Anthropic, GitHub, GitLab, Slack, Google, Stripe, SendGrid, npm, JWT, DB-connection-URI, PEM, Azure, and GCP service-account credentials — but has no pattern for PyPI API tokens, which every real PyPI/TestPyPI upload token begins with the fixed, well-known prefix pypi-AgEIcHlwaS5vcmc (base64 of pypi.org:, itself followed by more base64 — a very high-confidence, zero-ambiguity prefix, the same shape as the existing npm_/gh[pousr]_ entries). Add one new compiled pattern to _CREDENTIAL_PATTERNS for it, following the existing entries' style (a short comment naming the vendor).

Difficulty: level-7

Additive engine change: exactly one new line in an existing, well-understood pattern table, plus a test — genuinely small, but it's a new detection case in the objective guardrail's secret-exfiltration path, so it's classified with the other rule/detector-table additions rather than as a docs/mechanical change.

Safety constraints (do not weaken these)

• Purely additive: this adds one new match case to an existing high-confidence pattern list — it must never touch, loosen, or remove any existing pattern in _CREDENTIAL_PATTERNS, and must not change the rule's existing PASS/AUTH/BLOCK combination logic (secrets.py's own module docstring: exfil + external destination → BLOCK, local access → AUTH).
• Redaction: the test must confirm a synthetic PyPI token never appears in any explanation, reason, or log — only its keyed HMAC fingerprint (per doberman.storage.fingerprint.fingerprint, already used for every other pattern here).

Starting files

• src/doberman/engine/rules/secrets.py — _CREDENTIAL_PATTERNS (lines 52–83); add the new re.compile(...) entry with a # PyPI API token comment, matching the existing style.
• tests/unit/ — extend the existing secret-pattern test file for secrets.py (search for how the npm_ or gh[pousr]_ patterns are tested and mirror it) with a synthetic PyPI-token fixture.

Acceptance criteria

• [ ] A synthetic string starting with pypi-AgEIcHlwaS5vcmc is detected as secret material by the existing secret-detection path.
• [ ] A synthetic PyPI token never appears in any GuardrailResult.explanation, log line, or stored decision row — same redaction test pattern already used for the other credential types.
• [ ] No existing _CREDENTIAL_PATTERNS entry is modified or removed.
• [ ] Existing behaviour unchanged (no test weakened, no output format broken)
• [ ] ruff check . && ruff format --check . and lint-imports pass
• [ ] pytest passes

How to validate

pytest tests/unit -k secret -v

Out of scope

• Do not add detection for PyPI's username/password legacy auth (only the modern pypi- prefixed token format).
• Do not change how exfiltration vs. local-access is decided — only the pattern table.

Notes for contributors

New here? Start with CONTRIBUTING.md. Issues are labelled level-1 (docs only) through level-10 (new subsystem) — pick one at your level and climb. Comment to claim an issue before starting.

[Anonymous]

Originally posted by: Varshith-Kali

I'd like to work on this. Will add the PyPI API-token regex pattern to _CREDENTIAL_PATTERNS\ in \secrets.py\ with a corresponding test. Will open a PR shortly.

[Anonymous]

Originally posted by: fu351

Go ahead — it's yours. Thanks for picking it up, and sorry for the slow reply.

Your read of the code is right: _CREDENTIAL_PATTERNS is at src/doberman/engine/rules/secrets.py:52, and there's no PyPI pattern in there today.

Three things worth knowing before you start, because they're the difference between a pattern that gets merged and one that gets sent back:

1. False positives cost real user trust here. Every spurious match becomes an AUTH prompt interrupting someone's agent. Prompt fatigue is a security problem in its own right — a user who's been interrupted twenty times for nothing starts approving without reading. So please anchor the pattern tightly. PyPI tokens have a distinctive shape (a pypi- prefix followed by a long base64url macaroon payload), which makes this one of the easier credentials to match precisely — take advantage of that rather than matching anything token-shaped.

2. Ship a negative test alongside the positive one. The convention in this file is both directions:

• a synthetic token that must be detected, and
• a benign lookalike that must not trip it — e.g. a package named pypi-something, or prose mentioning pypi- in a comment or URL.

The negative test is the one reviewers care about most. We've had to tighten four over-eager patterns already for exactly this reason.

3. Use a clearly fake token. Never a real credential, even an expired or revoked one — the repo runs gitleaks over full history on every PR, and a real-looking token in a test fixture is the kind of thing that has to be treated as leaked and rotated. Make it obviously synthetic.

One framing note: secret detection is defense-in-depth, not a guarantee. A new pattern raises the floor; it doesn't make the rule airtight, and the docs shouldn't claim otherwise.

Shout if anything in secrets.py is unclear — happy to answer questions on this thread before you open the PR.

[Anonymous]

• assigned_to: Varshith-Kali

[Anonymous]

Originally posted by: fu351

Checking in, with no pressure attached.

It's been a few days, so I mostly want to be sure the ball isn't sitting in my court. If anything in secrets.py is unclear, or if the negative-test convention I described turns out fiddlier than it sounded, say so on this thread and I'll walk through it with you. That guidance took me a couple of minutes to type and is genuinely more work to satisfy, which is a bad trade I'd rather fix than leave sitting.

If you've moved on, or things simply got busy, that's completely fine. Say the word and I'll unassign it with no hard feelings and put it back in the pool.

There's also a Discord if a faster back-and-forth would help: https://discord.gg/Sfy5XGNqty. Questions tend to get answered there in minutes rather than days. Entirely optional, this thread works too.

No deadline on this either way.

[Anonymous]

Originally posted by: fu351

Checking in: still planning to take this one? No pressure and no deadline. If you have moved on, say the word and I will unassign it, no hard feelings at all. If you are stuck anywhere (where the secret-detection rule lives, the pattern itself, how to test it), ask right here and I will point you in. Discord works too if that is easier: https://discord.gg/Sfy5XGNqty

[Anonymous]

Originally posted by: fu351

Hi @Varshith-Kali — I'm closing this one out, and I want to be straight about why. You'd claimed it and I assigned it to you, but after a couple of check-ins here it stayed quiet, and in the meantime @tomatotomata opened a complete, passing PR for the same PyPI-token pattern (#319). Rather than let the work sit, I've gone with theirs.

No hard feelings at all — the timing just didn't line up, and that happens. I'd genuinely like to have you building here. If you want to jump back in, #227 (a dedicated test for RoleBoundaryRule's multi-path worst-wins loop, level-4) is a good next pick and sits right in the same engine area you were already reading. Say the word and it's yours.

[Anonymous]

Ticket changed by: fu351

• status: open --> closed

[Anonymous]

Originally posted by: fu351

@Varshith-Kali Thanks again for engaging on this one.

Quick community roll-call: are you in the Doberman Discord? If yes, drop your handle here (or DM me there) so I can match the GitHub side of your work to your Discord name. If not, you're warmly invited: https://discord.gg/Sfy5XGNqty