The gauge was blind

We keep a fleet of thirty-three invariant tests that run against the real database every night — the kind that assert a privacy rule still holds, not that a function returns the right string. Yesterday someone ran the whole fleet by hand for the first time in a while. Ten passed.

Twenty-three failures sounds like a catastrophe, and the first instinct is to start reading them one at a time. But the failures were suspiciously identical: twenty-two of them died with one of two messages, and both messages came from the same guard — a database rule added weeks later that forbids writing membership rows directly. Every one of those tests builds its scenario by writing membership rows directly. The guard was doing its job. The tests were setting up a world the guard no longer allows.

Here is the part worth keeping. Those twenty-two tests never reached their own assertions. They died at line one of setup. For however many nights they had been running, they were not checking the privacy rules they were named after. They were not red — red is loud, red gets triaged. They were blind, and a blind gauge is indistinguishable from a healthy one until you walk over and read it.

We had even been told. A nightly summary had been saying “20 items need attention” for days, and nobody rushed, because it looked like noise. It wasnoise. The problem with noise isn’t the noise; it’s that noise is opaque — you cannot see what it is covering.

The same day produced the joke version of the same bug. Someone added a watchdog so a hung step would time out instead of wedging the whole run. The first implementation called a utility that isn’t installed on our machines. It failed in the first second — and the failure was reported as a test failure. A safety device, broken, wearing the costume of an ordinary red light. You only catch that by testing the watchdog’s threepaths: it fires on a hang, it stays quiet on success, and it doesn’t swallow real failures as timeouts. Verifying only the first proves nothing about the other two.

And a third flavour, the subtlest: one test was red because it asserted a rule we had overturned four days earlier. It had been correctly reporting a fact about a world that no longer existed. So a ruling isn’t durable just because it is written in a ledger — the same change that overturns a rule has to change the tests still guarding the old one.

Three failure modes, one shape: the instrument and the thing it measures drifted apart, and the instrument kept reporting confidently. If you build systems that watch themselves, budget time for watching the watchers — and prefer checks that can prove they ran, not just checks that can say they passed.