#115 Add an optional fail-closed release-readiness review phase

[Anonymous]

Originally created by: yablokolabs

Summary

A generated repository can have a green self-generated test suite and still be unsafe or misleading to publish. Yabloko Labs' Atomic evaluation required three successive independent review cycles after the nominal coding pass; each found additional correctness, numerical, packaging, metadata, documentation, or provenance defects.

This issue consolidates findings §13, §16, and §17. It is not solely a runtime bug—model quality and task scope matter—but Atomic should not equate “generated tests pass” with “public-ready.”

Observed gap

The system prompt correctly requires real execution and verification:

• stable-prefix.ts (github.com)

That is necessary but insufficient for release claims. In the evaluated repository, passing tests coexisted with defects including:

• invalid citation/package metadata and placeholder URLs;
• incorrect licensing/provenance attribution;
• CI installing an undefined dependency extra;
• untested mathematical edge cases and non-finite inputs;
• non-standard JSON output;
• no-op tests and incomplete formula assertions;
• additional numerical overflow/underflow defects found only by adversarial review.

The final tree was publishable only after a frozen-hash numerical audit and an independent security/publication audit both passed.

Proposed behavior

Add an opt-in, fail-closed release-review workflow for requests that explicitly target publication, packaging, or release. It should separate implementation from review and bind the verdict to the exact reviewed tree/hash.

A useful baseline can include:

1. clean install/build from a fresh environment;
2. project tests plus warnings-as-errors where supported;
3. package/wheel/archive content inspection;
4. placeholder, secret, private-path, and agent-provenance scan;
5. metadata, license, citation, URLs, and documented-command validation;
6. an approved second reviewer focused on adversarial domain edge cases;
7. invalidation of the verdict after any file change.

Explicit publish intent is not consent to transmit a private repository to another provider. Cross-provider review or external hooks require a separate approval naming the reviewer/provider and data boundary.

Acceptance criteria

• Release review is opt-in or triggered only by explicit publish/release intent; normal coding tasks are not slowed down.
• The implementing model's generated tests are not the sole release evidence.
• Review defaults to a local reviewer or an already approved trust boundary.
• Before any source is sent to another provider or external hook, Atomic discloses what repository content will leave the machine and obtains explicit approval for that reviewer/provider and data boundary.
• External hooks run read-only and sandboxed with the minimum necessary repository scope.
• Unavailable external review reports review unavailable; it neither exports code silently nor self-certifies.
• The review records and reports the exact git/tree hash or workspace fingerprint.
• Any post-review mutation invalidates the PASS state.
• Failed checks are surfaced as publication blockers, not hidden behind an overall green test count.
• The final reply distinguishes “implementation complete,” “tests passing,” and “release-ready.”
• Documentation explains that Atomic assists with review but does not guarantee correctness or certification.