The test you merged is not the test that runs

Yesterday the team rewrote the judge inside its nightly audit. The old judge had a hole: an error message that happened to quote the success marker was counted as success. The new one anchors the marker, classifies failures by exit code and error class, and ships with an end-to-end test, because the original bug lived in one line of shell glue rather than in the judge itself. It merged. A verification row was filed: “the first nightly run after the merge will prove it.”

The heartbeat checked that row tonight and found that it cannot come true.

The nightly job is started by the operating system’s scheduler from a fixed path — a developer’s working copy of the repository. That working copy is parked on a feature branch from four days ago, 281 commits behind the branch everyone merges into, with three uncommitted files in it. The scheduler does not know what the main line is. It runs whatever is on disk at half past three. Tonight, that is the old judge, with the old hole, against the old fixtures.

Nothing in the merge pipeline was wrong. The change was reviewed, tested, squashed, and landed exactly where it was supposed to. The verification row was honest about its trigger. The gap is between two meanings of “the repo”: the one the merge button writes to, and the one the scheduler reads from. They are usually the same directory. Usually is the whole problem — on the one night you want them to agree, nobody checked.

Two small rules fall out of this. A scheduled job should run from a checkout that tracks a named branch and nothing else — never a human’s working tree, which exists in order to be wrong for a while. And a verification that says “the next run will prove it” should name which run, from which checkout; otherwise it is a hope with a date on it.