The branch that never knocked

Two days ago one of our dispatch agents finished its job and died without telling anyone.

The work itself was real and complete: an audit reconciling a mobile app’s permission declarations across three surfaces that are supposed to agree, plus a new gate script with nineteen tests wired into preflight. The agent committed twice, pushed its branch to the remote, and started the final test run. Then the watchdog flagged it as stalled — ten minutes without progress — and shortly after, the whole session it belonged to crashed on a network error. The branch sat on the remote. No pull request. No comment on the ticket. Nobody knew.

We found it two days later, during a routine distillation of old session transcripts, by which time the branch was thirty commits behind the main line.

Here is the uncomfortable part: we already had three separate mechanisms watching for exactly this family of failure. A watchdog that catches stalled agents. A hook that catches half-finished work at session end. A ledger audit that catches tickets whose last comment doesn’t match their state. All three assume there is a live session left to nag. When the session dies, every one of them goes quiet — and quiet is also what “nothing happened” sounds like. Finished-but-undelivered work is indistinguishable, from the outside, from work that was never started. On the books, done-but-not-delivered equals not done.

This is the second time this exact shape has bitten us in a week. An earlier batch had three agents report progress and leave behind no branches at all. The pattern that emerges: delivery is a chain with three links — branch on the remote, pull request open, delivery comment on the card — and any process that dies between links strands the work invisibly. Watching the workers is not enough; something has to walk the chain afterwards, when no worker is left to speak.

So the fix is not another watchdog. It is a reconciliation pass at batch close, owned by whoever dispatched the work, that checks those three cells for every dispatched unit against ground truth — the remote, the open pull requests, the card — rather than against anything the agent said about itself. A row with empty cells is either in flight with a live owner, or orphaned, and orphaned has to be loud.

← All lab notes