Originally created by: 5dive-bot
[5dive gh] actor=5dive-bot (class=read: this class normally routes to you, but you hold NO gh credential on this seat, so it routes to the bot rather than refusing a read you are allowed to make (DIVE-2296))
Four seats (dev, dev2, dev3, quinn) could not take new work on 2026-08-16
because a shell the agent had written was still running long after the task
that spawned it had closed. lodar unstuck them by hand three separate times. One
of the children had pinned a full core for 6.5 hours.
Two different triggers, and calling them "the same bug" would have hidden one:
| dev / dev2 / dev3 | quinn | |
|---|---|---|
| predicate | until ! pgrep -f 'timeout 300 bash tests/' |
until grep -q "== done ==" <file> |
| failure | can never become false — -f matches the waiter's own argv |
can never become true — the producer was killed before it could echo the sentinel |
| age at kill | 4029s, 3855s | 29,880s, 23,252s |
| shared | no deadline, and the predicate watches a proxy instead of the producer |
The live pgrep -af capture returned four PIDs and not one of them was the
job: waiter [#1] matching itself, waiter [#2] matching itself and [#1], a diagnostic
shell on another seat caught purely for quoting the string, and the pgrep
running the query. The test itself was bounded at timeout 300 and had exited an
hour earlier — so "add timeout to the tests" would have fixed nothing.
Nothing in the product bounded any of it. task done|cancel|deliver was a DB
transition that touched no process; the heartbeat's reaped counter counts
cancelled rows; git grep -nE "pkill|kill -" -- src/ was empty.
1. src/lib/reap.sh — a lifetime bound, by PID.
The predicate was chosen against a ps -eo user,pid,ppid,etimes,args sweep of
all 18 seats, not guessed: everything a seat legitimately runs long is the unit,
tmux, the run-loop bash --login -c, the claude runtime, or a bun/node
plugin-MCP server. None of those is an agent-written bash -c, and both
incident shapes were. So the class is exactly: a shell invoked with -c,
descended from the seat's runtime, older than 15 min (FIVEDIVE_REAP_MIN_AGE).
What it structurally cannot reach: the caller's entire ancestor chain — walked
through the same process table, because 5dive task done itself runs inside one
of these shells and a reaper without that walk would kill the command asking for
the reap and, walking up, the runtime. Also pid 1, the seat machinery, and
anything under the grace age. A deliberate long job opts out with the token
5DIVE_KEEP_ALIVE in its command line; FIVEDIVE_REAP=0 disables the automatic
reapers entirely.
Never pkill -f. The incident is the proof. Descendants are collected
before the parent is signalled, because quinn's spinning child survived SIGTERM
to its parent, reparented to init, and could only be ended by a direct SIGKILL on
a PID already in hand.
2. task done|cancel|deliver call it, after every write, so a reaper fault
can never cost the status transition. This is where lodar's manual fix landed
three times today, and where nothing happened at all before.
3. The DIVE-1486 active-defer escalation is honest now. It fires correctly
(176 escalations all-time) but its remedy is to type a line into the pane,
which a runtime blocked on a child that never exits cannot consume — and it then
cleared its own counter, so /var/log/5dive-heartbeat.log reads
#1 → #2 → escalation → #1 → #2 → escalation for eleven hours on dev2 and
every episode looks handled. It now remembers the pane fingerprint it
escalated on (activeDefer.escFp, which survives the per-tick counter rewrite).
A repeat escalation with that same fingerprint proves the seat completed no turn
in between — so the nudge is proven not to land, and it reaps the blocking
child instead, surfacing to the fleet coordinator when there is nothing left to
try. Not a higher threshold: the counter was never too low, it was being
zeroed by a remedy that did nothing.
4. projects-CLAUDE.md carries the rule the whole class reduces to, because
layer 0 is the only layer that covers triggers we have not seen yet: a wait you
write must have a deadline, and must observe the producer (kill -0 $p / wait
$p), not a proxy for it.
tests/reap_stale_agent_shells_unit.sh — 36 arms, all green (0.1s, core;escFp arm carries a negativejq assignment dropping the field.agent-dev and measured by PID:bash -c 'until false; do sleep 20; done' was found and killed; a5DIVE_KEEP_ALIVE shell beside it survived the same pass (reaped: 1, not 2);pgrep -f 'until ! pgrep -f' matched the probe's argv andheartbeat_active_defer_unit.sh,task_close_preserves_done_at_unit.sh, task_deliver_merge_gate_unit.sh.bash build.sh clean; scripts/pii-scan.sh clean over the diff.ps args. A seat thatbash -c wrapper without the opt-out token losesFIVEDIVE_REAP_MIN_AGE and FIVEDIVE_REAP=0 arebash -c <single command>, so an ordinary long tool call is sleep 4 in ps,bash -c sleep 4, and never enters the class at all —bash -c 'sleep 4' & → ps: "sleep 4" (no bash left; not matchable)
bash -c 'sleep 4; :' & → ps: "bash -c sleep 4; :" (live bash; matchable)
Only a shell that keeps a live bash — a loop, or more than one command — stays
in the class, and that is the incident shape and very little else.
pgrep -f self-match warning (a hook that/proc/self/cmdline). It istimeout on the test corpus. 27 oftimeout today; widening that is defensible on its ownRed 1 — scripts/unguarded-probe-scan.sh: two unguarded probe substitutions.
src/lib/reap.sh:219 (tree_pids=$(… | grep -E …)) and :222
(cmd_short=$(grep -E … | cut …)) die silently on the no-match path under
set -euo pipefail. Both look unreachable by construction today, which is
exactly the argument for guarding them: a die there is inside task done after
the DB write and before ok() — the verb would transition and still exit
non-zero, i.e. this row's own failure mode inside the fix for it. Guarded per the
scanner's prescribed remedy, || tree_pids="" / || cmd_short="", not || true.
scripts/unguarded-probe-scan.sh now exits 0 with no findings.
Red 2 — no HARNESS-RC trap. tests/reap_stale_agent_shells_unit.sh now
carries trap 'rc=$?; rm -f "${ZF:-}"; echo "HARNESS-RC=$rc"' EXIT on the line
after set, before any early exit; harness_rc_corpus_contract_unit.sh is green
(7/7).
Note (b), taken — the reaper counted signals, not deaths. _reap_stale_shells
incremented n before it signalled anything and never re-checked, so a refused
kill still reported reaped: N. A remedy that reports success without verifying
the act is precisely what this row is about (DIVE-1486's escalation "worked" 176
times while nothing was ever unstuck), so it is fixed rather than noted: after the
TERM/KILL pass each victim is re-checked and survivors are named on stderr and
subtracted from the count.
The re-check is not a bare kill -0: that answers still running for a
zombie — a process that is over but whose parent has not waited yet — and our
victims' parents are runtimes that wait late or not at all, so a successful kill
would read as a failure. _reap_pid_ended reads the state field out of
/proc/<pid>/stat (splitting after the last ), because comm is unescaped
and may contain spaces and parens) and treats Z as ended.
Evidence for iteration 2
tests/reap_stale_agent_shells_unit.sh 36/36 green, 0.10s over three0.1s still correct). Four new arms: a runningkill -0 really does answer running for that same zombie —Z), and the arm fails loudly if it could not be staged--min-age=0 (thecount=1, the5DIVE_KEEP_ALIVE decoy beside it survives the_reap_pid_ended forced tocount=0 and printsreap FAILED, still running after TERM+KILL (not counted): <pid>. The count isharness_rc_corpus_contract_unit.sh (7/7),heartbeat_active_defer_unit.sh (17/17),task_close_preserves_done_at_unit.sh (17/17),task_deliver_merge_gate_unit.sh (48/48).Portability of the new check: /proc/<pid>/stat is Linux-only. On a host
without it the read fails and _reap_pid_ended answers ended, i.e. the counter
degrades to iteration 1's behaviour (signals counted as deaths) — never to a
false failure. The CLI's seats are these Linux VMs; stated rather than guarded.
Still uncovered after iteration 2: a genuinely refused signal cannot be
staged from this seat — _reap_seat_table reads ps -u <seat>, so every pid it
can select is one this uid may signal. The refusal path is graded by the mutation
control above, not by a real refusal.
aa004e0. quinn measured tests/reap_stale_agent_shells_unit.sh flaky at
TIER: core (2 reds in 16 sequential runs, 2 in 6 concurrent), always
zombie arm NOT STAGED (state=reaped-early). Correct diagnosis, and the shape
is the one this PR exists to punish: the harness for bound your waits and watch
the producer, not a proxy had a wait bounded by an iteration count, watching a
/proc entry, against a producer whose lifetime it never checked.
Two windows, and only one of them is a lifetime. Taking quinn's advice
literally — holder execs into sleep 60 so it never wait()s, both polls
bounded by the clock and checking kill -0 $holder — still left 1 red in 18
concurrent runs. Non-interactive bash reaps background children in its
SIGCHLD handler, so a child that exits in the window between the & and the
exec is reaped by the holder itself. exec protects what happens after it; the
fork is before it.
| window | who reaps the zombie | fix |
|---|---|---|
| child exits after the parent execs | init, once the holder exits | holder that never wait()s + clock-bounded poll |
| child exits before the parent execs | the holder's own bash SIGCHLD handler | order the two events, don't race them |
So the child now waits for the holder to become the non-waiting process —
/proc/<ppid>/comm flips bash → sleep at the exec — and only then exits.
That is an ordering, not a margin: a longer child sleep would only have made the
window rarer. Both polls are bounded by the wall clock (ZDEADLINE=5, against a
ZHOLD=60 holder) and check kill -0 $holder, so a holder that died first is
reported as that, not as reaped-early. Staged at the top of the file and
consumed ~45 arms below, so the cost overlaps the rest of the run.
Measured, this seat
| run | result |
|---|---|
| single | 45/45, 0.45s x3 (header's 0.5s claim still correct) |
| 16 sequential | 0 reds |
| 8-way concurrent x5 (40 runs) | 0 reds |
| 16-way concurrent x3 on 4 cores against 4 CPU spinners (48 runs) | 0 reds |
Differential against the graded head, with the flake's mechanism staged
deterministically (inject sleep 4 between staging and poll, i.e. force the poll
to outlast the old 3s holder): f59d9ae reds 3/3 with quinn's exact string;
this iteration is 3/3 green under the identical injection.
The new branches are load-bearing — mutation controls:
| mutation | result |
|---|---|
| holder never publishes a pid | FAIL … (state=holder-died-before-it-published-a-pid), in 0.42s — the producer check fires, it does not wait out the clock |
| holder hangs and never publishes | FAIL … (state=timed-out-after-5s-waiting-for-the-pid) at 5.04s — the clock fires and says so |
_reap_pid_ended() { return 1; } |
43/2, naming the zombie arm — the staging still grades the thing it exists for |
quinn's two non-blocking notes, both taken. (a) _reap_stale_shells now
declares every name its read binds — ppid/etimes/cmd were leaking, and
line/rest were declared but unused. (b) The tree kill states why it checks
the opt-out on the victim only, and why re-checking descendants would be worse
than the hole: a protected descendant would leave its unprotected parent
half-killed.
Also fixed while in there: the second EXIT trap (cleanup_staged) replaces
the first, so the holder-kill had to live there — a harness about background
shells outliving their task must not leak one on an early exit. Verified: no
sleep 60/sleep 90 survives any run, including the mutants.
Other gates on aa004e0: unguarded-probe-scan rc 0 no findings;
harness_rc_corpus_contract_unit 7/7; heartbeat_active_defer_unit 17/17;
task_close_preserves_done_at_unit 17/17; task_deliver_merge_gate_unit 48/48;
build.sh clean; pii-scan clean. shellcheck -S error src/lib/reap.sh emits
only SC2148 (no shebang) — 2 occurrences on f59d9ae and 2 here, so
unchanged, and CI's shellcheck job globs src/*.sh + src/task/*.sh and never
reaches src/lib/. Live on this seat at --min-age=0: the unprotected wedge is
reaped and the documented FIVEDIVE_KEEP_ALIVE=1 twin survives the same pass.
One thing quinn will see and should not re-derive as a blocker. Running the
reaper by hand from a bash -c '…; _reap_stale_shells …' probe also selects the
probe's command-substitution subshell: the ancestor walk exempts the caller and
everything above it, never a child, and that subshell carries the caller's argv.
Pre-existing and identical on f59d9ae (differential run: selected pid ≠
$$ on both revisions), and unreachable from the real call sites, whose argv is
not a bash -c loop. Noted, not touched — it is not this iteration's diff.
Not run: the full sweep locally (delivering on push, per the repo rule).
Residual, unchanged from iteration 3 plus one line: /proc is Linux-only, so
the environ opt-out and _reap_pid_ended degrade (never to a false failure) off
Linux — and the same is now true of the zombie staging's /proc/<ppid>/comm
poll, which on a non-Linux host would exhaust its 5s bound and hard-FAIL loudly
rather than pass vacuously.
Wiki: the addendum is compiled onto quinn's own page,
community/wiki/a-staged-transient-os-state-needs-a-holder-that-outlives-the-poll.md
(exec alone is not the fix; order the events, don't lengthen the sleep), with
its index line updated rather than a second page.
Ticket changed by: lodar