#278 [Release] Fail-closed artifact validation: an incomplete build must be unable to ship

[Anonymous]

Originally created by: Avazbek22

Summary

A build that lost part of its payload must be physically unable to ship. Not "detected later", not "reported nicely" — unable to leave the pipeline, on every distribution channel we have and every channel we add.

The Store 5.1 incident proves the gap: the MSIX shipped without loadable Tree-sitter grammars, three user-facing features silently died for every Store user, and release validation was green — it checked the manifest, the languages, the execution alias, but never asked whether the package could actually do what the product promises. The fix added one gate (Store packages must carry the grammar payload). One gate for one payload in one channel is not a strategy.

What can silently fall out of a build

Payload classes, each of which is assembled at pack/publish time and can be lost by a packaging change without any compile error:

• Native libraries — Tree-sitter core, per-language grammars, any future native component. Delivery is channel-specific by design (embedded for portable builds, loose signed content for MSIX), so every channel is its own chance to lose them.
• Asset payloads — localization satellites (the Store build once lost 9 of them — a sibling incident, caught by hand too), syntax-highlighting definitions, icons, help content.
• Layout contracts — execution alias, licenses and third-party notices, anything a runtime locator resolves by path.

Channels, each with its own packaging pipeline: GitHub self-contained artifacts, Microsoft Store MSIX, winget (repackaged GitHub), and next the NuGet tool packages (#276) and npm platform packages (#277). The payload × channel matrix only grows.

Why the existing safety net missed it

1. Validation checked structure, not capability. The release script asserted manifest contracts and file presence for a few known paths — nothing verified that a feature-critical payload was inside, and nothing ran the artifact.
2. The right test existed but watched the wrong build. grammar-delivery.yml + GrammarDeliveryProbe verify content-mode delivery thoroughly — against their own parallel publish, not against the artifact that ships. A green parallel build proved nothing about the MSIX.
3. Expected contents have no single source of truth. The curated grammar list lives in Infrastructure.csproj; the new Store gate re-hardcodes "≥ 20 plus two sentinels". Two copies of the truth drift, and every new payload or channel invites a third.
4. Unit and integration suites run against dev builds, where embedded delivery always works — no suite ever executes the packaged layout.

Proposal

1. Payload manifest as the single source of truth. The build exports the expected-contents list per artifact kind (which grammar libraries, which satellites, which content files, per RID). Validators consume it; nothing re-hardcodes counts or names. The existing Store grammar gate migrates onto the manifest instead of its inline "≥ 20".
2. Static completeness gate for every artifact. Release validation opens each artifact it is about to publish — GitHub archive, each platform MSIX, later each NuGet RID package and npm platform package — and diffs actual contents against the manifest. Any missing entry fails the release. This generalizes the gate added after the incident from one payload/one channel to all payloads/all channels.
3. Functional smoke of the shipped artifact, not the dev build. For each publishable artifact, run the real binary and exercise one capability-critical operation per payload class end-to-end: compress a known source file and assert the output changed; switch to a non-default language and assert a localized string; run a secret-findings check and assert a detection. The Store incident is exactly the failure this catches: a green static check with a dead runtime. Where the artifact cannot be executed in the pipeline (MSIX certification constraints), run the probe against the extracted package layout — the layout the runtime locator actually resolves against.
4. Channel onboarding contract. A new distribution channel cannot ship without its completeness gate and smoke hook. [#276] and [#277] must include these gates in their acceptance criteria before their first publish.
5. Run gates early, not only at release. PRs that touch packaging, payload wiring, or delivery properties trigger the artifact validation in CI (the grammar-delivery.yml trigger model), so a payload regression is caught at review time, not at release time.

Non-goals

• Runtime degradation notices (telling the user a capability is unavailable) are a separate concern and intentionally out of scope here — this issue is about never publishing such a build in the first place.
• No slowdown of inner-loop dev builds and test runs; gates attach to packaging pipelines and packaging-touching PRs.

Acceptance criteria

• Deleting any single manifest-listed file from any publishable artifact makes the release fail, demonstrably (a deliberate mutation test in CI proves the gate bites).
• No validator carries its own copy of expected contents — the Store grammar gate reads the manifest like everyone else.
• Each channel's pipeline runs a functional smoke against the artifact it publishes; the compression smoke on the packaged layout would have failed the 5.1 Store release.
• Adding a channel without a gate is impossible to do quietly: the channel checklist is documented, and [#276]/#277 reference it.