#194 tests: wire a real (non-mocked) plugin install into CI

[Anonymous]

Originally created by: fu351
Originally owned by: jasperdingg

What

Doberman's split-feature design relies on Python entry-point discovery: core defines interfaces (RULE_GROUP/DETECTOR_GROUP in src/doberman/engine/registry.py), plugins register implementations. Today two things exist but neither closes the loop: tests/unit/test_registry.py proves discovery works against a hand-rolled fake entry_points() (monkeypatched, no real install), and examples/plugin-guardrail/tests/test_example_rule.py::test_entry_point_is_discoverable_after_install proves it against a really-installed package — but that test only runs when a contributor manually does pip install -e examples/plugin-guardrail first, and is not part of the default CI pytest run (CI's testpaths is ["tests", "tools"]; it never installs or runs anything under examples/). So CI never automatically proves real (non-monkeypatched) entry-point discovery end-to-end. Add an integration test under tests/integration/ that installs a real plugin package as part of the test itself and asserts discover_rules() finds it — wired into the default CI run.

Difficulty: level-5

The mechanics (subprocess.run(["pip", "install", "-e", ...])) are simple, but correctness here is about isolation: the test must clean up (uninstall) reliably even on failure, and must not leave the plugin installed for later tests — in particular the "no plugins installed" default-discovery guarantees exercised elsewhere in the suite. Reusing examples/plugin-guardrail as the fixture avoids inventing a second demo package.

Safety constraints (do not weaken these)

• Plugin isolation: a broken/hostile plugin must still be defensively skipped, never crash the registry — this test proves the happy path, it must not weaken discover_rules()'s existing defensive-loading behaviour.
• Must not leak an installed plugin across test runs and break any test elsewhere in the suite that expects discover_rules() == [] with nothing installed.

Starting files

• src/doberman/engine/registry.py — discover_rules(), RULE_GROUP.
• examples/plugin-guardrail/ — the real installable fixture to reuse (pip install -e examples/plugin-guardrail).
• tests/unit/test_registry.py — the existing monkeypatched-fake coverage; read it first so this issue doesn't duplicate it.
• New test in tests/integration/ (e.g. test_real_plugin_install_discovery.py).

Acceptance criteria

• [ ] New integration test installs examples/plugin-guardrail for real (via subprocess/pip) and asserts discover_rules() returns an ExampleRule instance.
• [ ] The plugin is uninstalled at the end of the test (including on failure — use a fixture teardown, not a bare end-of-test call).
• [ ] Running the full suite twice in a row (pytest then pytest again) leaves no residual installed plugin.
• [ ] Existing behaviour unchanged (no test weakened, no output format broken)
• [ ] ruff check . && ruff format --check . and lint-imports pass
• [ ] pytest passes

How to validate

pytest tests/integration/test_real_plugin_install_discovery.py -v
pip show doberman-example-plugin-guardrail  # should report "not found" after the test run

Out of scope

• Do not add examples/ to CI's default testpaths — keep the new test self-contained under tests/integration/.
• Do not modify examples/plugin-guardrail/ itself.

Notes for contributors

New here? Start with CONTRIBUTING.md. Issues are labelled level-1 (docs only) through level-10 (new subsystem) — pick one at your level and climb. Comment to claim an issue before starting.

[Anonymous]

Originally posted by: jasperdingg

I'd love to take this one, will open a PR soon!

[Anonymous]

• assigned_to: jasperdingg

[Anonymous]

Originally posted by: fu351

Yours — assigned. Apologies for the delay in answering.

This is the one I suggested when merging [#221], and I think it suits you specifically. In that PR you passed load_plugins=False so an installed plugin couldn't perturb the sweep — that instinct, distrusting what the environment might inject, is exactly what this issue needs turned around: right now the plugin/entry-point discovery path is only exercised against mocks, so nothing proves a genuinely installed plugin package actually registers and runs.

Two pieces of context that should save you time:

• There's already a worked example to install rather than a fixture to invent: examples/plugin-guardrail/ is a tutorial plugin package that registers through the doberman.rules entry-point group. Installing that in CI and asserting its rule shows up and fires is the shortest path.
• CI just gained a package-smoke-test job (#220) that builds the wheel and installs it into a clean venv. That's a good pattern to borrow, and possibly a good place to hang this — an installed-plugin check has the same "does it work outside the source tree?" shape. Your call whether it belongs in that job or its own.

The failure mode to design against: an entry-point test that passes because the plugin was importable from the source tree rather than because it was installed and discovered. Same class of bug as [#220]'s original gap, where installing the wheel proved the modules shipped but never proved the package data did. Worth making the test fail if entry-point discovery is disabled.

No rush on ordering — [#175] is the quicker of your two if you'd rather clear that first.

[Anonymous]

Originally posted by: fu351

Delivered by [#318] (merged). Thanks @jasperdingg!

[Anonymous]

Ticket changed by: fu351

• status: open --> closed