The receipt was written first

Our company brain publishes a daily report. For two days it didn’t, and nobody noticed. The way we found out was a founder saying “feels like we haven’t had a daily report in a while.”

That sentence is the actual incident. Everything downstream of it is just mechanism. The mechanism, though, is worth writing down, because it is the kind of bug that looks like three small conveniences until the day they line up.

The scheduler runs at most once per calendar day. To enforce that, it writes today’s date to a stamp file, then does the work. Written in that order, the stamp does not mean “today’s run finished.” It means “today’s run was attempted.” The moment the process starts, the day is spent.

The second convenience: the script ran under strict shell settings, where any command exiting non-zero aborts immediately. So when the agent exited with an error — a model usage limit, twice, on two consecutive nights — the script didn’t log a failure, didn’t close its ledger entry, didn’t leave a marker. It stopped mid-sentence.

Put those together and you get a system that claims the day, dies a second later, declines to retry because the day is claimed, and tells nobody. Not a crash. A quiet, well-behaved, fully deterministic nothing.

The third part is the one I find hardest to defend. The evidence was sitting there the whole time. The run ledger had two orphaned start events with no matching end — one for each missing night, timestamped, in a file we wrote specifically so that runs would be auditable. Nothing read it. An audit trail that nobody and nothing reads is not an audit trail; it is a diary.

The fix is small and mostly consists of reversing an order. Let the script survive the agent’s failure instead of dying with it. Move the stamp to the end, and write it only on success — re-entrancy was already handled by a separate lock, so the stamp was free to mean the honest thing: this cycle actually completed. On failure, leave a marker and a loud line, and let the existing retry interval do what it was always able to do.

Two smaller things fell out of it. The ledger’s own lines turned out to be invalid JSON — paths containing quotes and non-ASCII characters had been interpolated into a JSON string without escaping, so the “machine-auditable trail” could not be parsed by a machine. And the stale stamp from the failed run was still on disk, silently guaranteeing that the very next cycle would skip too. Deleting it was part of the fix, not cleanup after it.

We verified by making the failures happen: a copy of the script in a scratch directory, a fake agent that exits non-zero, then one that exits clean. The failure path leaves no stamp, a failure marker and a loud line. The success path leaves a stamp. Every ledger line parses. Reading the diff would have told us the same story, and reading the diff is how the original order got written in the first place.

The rule I’d keep: a completion marker written before the work is not a completion marker, it’s a reservation. And any system whose failure mode is silence needs someone whose job is to look at the silence — otherwise the first alarm is a human’s vague feeling, days late.