#213 storage: add a configurable prune policy for the local decision log

[Anonymous]

Originally created by: fu351
Originally owned by: slegarraga

What

The local SQLite decision log (src/doberman/storage/log.py, the decisions table, schema owned by src/doberman/storage/db.py) grows without bound — one row per engine decision. Add a configurable max-age and/or max-row prune for fully-resolved rows, with a safe default (e.g. off, or a generous retention window).

This is a distinct module from the append-only policy-drift ledger (src/doberman/policy/drift.py, the policy_changes table) — that ledger is a separate tamper-evidence record for policy changes and is explicitly out of scope here (see safety constraints).

Difficulty: level-5

Additive storage-policy change — pruning never rewrites what already happened, but it touches the decision-log write/read path and needs a design that keeps pruning off the hot decision path.

Starting files

• src/doberman/storage/log.py — decision-log writer (decisions table)
• src/doberman/storage/db.py — schema (decisions table)
• src/doberman/policy/drift.py — the append-only policy-drift ledger (policy_changes table) — do not touch
• tests/integration/test_decision_log.py — existing decision-log test pattern

Acceptance criteria

• [ ] Configurable max-age and/or max-row prune for the decisions table, with a safe default
• [ ] Pruning only removes fully-resolved rows (never unresolved/pending)
• [ ] Pruning runs out-of-band from the decision path (not inline in a verdict)
• [ ] One test proves a decision's verdict is unaffected whether a prune is due or not
• [ ] One test proves old rows are actually gone past the threshold
• [ ] existing behaviour unchanged
• [ ] ruff check . && ruff format --check . and lint-imports pass
• [ ] pytest passes

How to validate

pip install -e ".[dev]"
ruff check . && ruff format --check .
lint-imports
pytest tests/integration/test_decision_log.py -v
pytest --cov=doberman --cov-report=term-missing

Out of scope

• The append-only policy-drift ledger (src/doberman/policy/drift.py, policy_changes table) — never prune it
• Un-redacting or exporting pruned data anywhere

Safety constraints (do not weaken these)

1. Logging never alters, blocks, or delays a decision — pruning must run out-of-band from the hot decision path, never inline in a verdict.
2. Do not prune the append-only policy-drift ledger (src/doberman/policy/drift.py, table policy_changes) — that ledger is a tamper-evidence record; pruning it defeats its purpose. This issue covers only the decision log (src/doberman/storage/log.py, table decisions).
3. Never prune unresolved/pending rows.
4. Pruning must not un-redact or export anything — it only deletes already-redacted rows past the threshold.

Notes for contributors

New here? Start with CONTRIBUTING.md. Issues are labelled level-1 (docs only) through level-10 (new subsystem) — pick one at your level and climb. Comment to claim an issue before starting.

[Anonymous]

Originally posted by: fu351

Re-leveled 7 → 5. A retention/prune policy on the decisions table is a config knob plus a bounded delete query and its tests — the same weight as the memory prune that already shipped.

[Anonymous]

Originally posted by: slegarraga

Hi @fu351! I'd love to take this one — thanks for the invitation in the merge comments of [#450], and for the kind words there. The storage readers I touched for that fix are exactly where this prune policy will live, so it feels like the natural next step.

My plan, roughly:

1. Read storage/log.py, storage/db.py, and the existing memory prune implementation you mentioned to match its config-knob + bounded-delete pattern.
2. Add configurable max-age and/or max-rows pruning for fully-resolved rows only, with a safe default (off by default or a generous window — happy to follow whatever the repo prefers).
3. Keep pruning strictly out-of-band from the hot decision path per the safety constraints.
4. Tests: verdict unaffected whether a prune is due, old rows actually gone past the threshold, existing behavior unchanged.

I'll comment here with any design questions before opening a PR. Thanks again for running such a welcoming project!

[Anonymous]

• assigned_to: slegarraga

[Anonymous]

Originally posted by: fu351

Thanks for taking this on Sebastian! If you have any questions that require some discussion, I suggest using our discord for easier comms.
https://discord.gg/Sfy5XGNqty

[Anonymous]

Originally posted by: slegarraga

Thank you, @fu351! That is really kind. I will start with a focused proposal for the configurable prune policy and keep the reader semantics from the recent fixes intact. If anything needs a quick discussion before code, I will bring it to Discord.

[Anonymous]

Originally posted by: slegarraga

Update: the focused implementation is ready for review in [#461]. It supports both max-age and max-row pruning, protects pending AUTH rows, stays completely outside the decision path, and leaves policy_changes untouched. Thanks again for the guidance and for such a friendly project to contribute to!

[Anonymous]

Ticket changed by: vinayjagan03

• status: open --> closed