Menu ▾ ▴

#527 fix(ci): the over-budget red stops asserting a cause it cannot measure (DIVE-2829)

closed
nobody
None
2026-08-10
2026-08-07
Anonymous
No

Originally created by: 5dive-bot

DIVE-2829 — the over-budget red stops asserting a cause it cannot measure

What this is: the message fix on the budget verdict's red path (scripts/run-harnesses.sh) plus 9 new mutation-graded arms (74-82) in tests/corpus_tier_budget_unit.sh.

Why: on 2026-08-05 the budget verdict red'd main off a SINGLE runner (416s/138%) and froze release-cut for ~40 minutes; a re-run of the same sha on a fresh runner read 245s/81% and went green. The failure text asserted "the CORPUS no longer fits its cap" — a cause this verdict cannot establish, since it has not excluded a slow runner — and sent readers to merge/retire/demote real coverage in response to a slow box. That is the expensive direction to be wrong in.

The row's recommended remedy is NOT implemented, because CI logs falsify its premise: on run 31051177868, attempt 1 (416s/138%) and attempt 2 (245s/81%) BOTH printed AGREES from pre probes 93%/89% that both clamped to the floor, and the SLOW attempt's post probe read FASTER (-10%). Promoting the post-corpus discriminator would not have fired on the very run it was designed for. Full measurement set is in the DIVE-2829 result.

What the diff does instead:

  • the over-budget red now prints WHAT IS MEASURED / WHAT IS NOT EXCLUDED and names a same-sha re-run on a DIFFERENT runner as the first action
  • the "measured TWICE ... not variance" claim is scoped to what it actually excludes (within-runner noise)
  • the "the corpus and not the VM" claim is scoped to the conditions that earn it
  • M1-M5 each restore one retired sentence and break exactly one named arm; M6 wires the row's own remedy into the exit ladder and breaks pre-existing arm 65, so the "post probe grades nothing" invariant is still guarded

Checked: tests/corpus_tier_budget_unit.sh 104 pass / 0 fail locally, re-run after the rebase onto a0190a2 (the two upstream commits, DIVE-2853/#524 and DIVE-2803/#522, touch no harness files); 4 neighbouring harnesses green (see task result).

NOT claimed:

  • the verdict is still not robust to runner speed and cannot be made so from inside one job — no in-job signal separates a slow box from a grown corpus on the measured pair. The acceptance's slow-runner arm needs a second RUNNER; that is a workflow/policy call, not this diff's.
  • no verdict semantics moved: exit codes, caps and scaling are unchanged, and the 300s cap is NOT raised.

Gate: human approval gate on DIVE-2829, answered approved 2026-08-07 15:07Z (signed closure verified).

Related

Tickets: #548

Discussion

  • Anonymous

    Anonymous - 2026-08-10

    Originally posted by: 5dive-bot

    SUPERSEDED — the earlier DO-NOT-MERGE on this comment pointed at the wrong risk. Merging is now the PREREQUISITE, not the hazard.

    Marcus ran the control from an admin seat and it answered the open question: an if:-skipped job DOES create a check-run with conclusion: skipped, and that satisfies a required context. Three instruments agreed on this PR (gh pr checks --required listed all 9 with the confirm rows reading skipping; mergeable_state stayed unstable, never blocked, stable across two reads; commits/6bdc9c6/check-runs carried both names as skipped).

    What the control could not see is why the order matters: #527 is the only branch where these jobs exist to be skipped. On every other branch they are ABSENT, no check-run is ever created, and a required context that never reports blocks forever — PR [#554] went clean to blocked on one API call with all its own checks green. Present-and-skipped and absent are indistinguishable in the YAML; only commits/<sha>/check-runs per branch tells them apart. Protection has been reverted and is byte-identical to baseline.

    Corrected order: deliver to main2 for grading on the code, merge this PR so the jobs exist on main, THEN add test-confirm / test-installed-host-confirm to required_status_checks.contexts (Marcus, admin seat) — after which every still-open PR must rebase before it can merge.

    Until step 3 lands the confirm verdict still cannot gate, so DIVE-2829 does not close on this merge. Residual, recorded rather than solved: enforce_admins is false, so the gate binds bot-routed merges and asks politely of an admin-credentialed one.

     

    Related

    Tickets: #554

  • Anonymous

    Anonymous - 2026-08-10

    Ticket changed by: 5dive-bot

    • status: open --> closed
     

Log in to post a comment.