The bugs your tests cannot see

We had a good day of shipping and a bad day of discovering. Nine pull requests merged. Also: an age gate that only existed on the phone, a privacy policy claiming a feature wasn’t built three days after it shipped, an error message hard-coded in the wrong language, a document telling every new teammate that a file was the security boundary when it wasn’t, nine tickets sitting open on work that was already done, and a backup file that was itself corrupt — so the documented recovery procedure restored the very state it was supposed to undo.

At the end of the day someone asked the obvious question: what should we audit next, so we find these before a human stumbles into them? Before answering it, we did something more useful — we went back and asked how each one had actually been found. The list was uncomfortable: by using the product in the real world; by looking at one screenshot; by reading the code and asking “who enforces this?”; by holding a document next to the code; by two documents contradicting each other; by comparing what a ticket claimed against version control; by following our own recovery steps and watching them fail. Not one of them was caught by a test. And the codebase is not untested — there are over four hundred and fifty of them, and they are good ones.

That is not a gap in coverage. Coverage is the wrong frame entirely, because every one of these bugs has the same shape: the same fact was stated in two places, and the two statements drifted apart. Code versus documentation. Code versus ticket. Client versus server. One language versus the other. App versus marketing site. Backup versus original. In every case both sides were individually consistent — which is exactly why the tests passed. A unit test verifies that a function agrees with itself. It has no opinion about whether your privacy policy agrees with your migrations.

This reframes what to build. The instinct is to write more tests, and it is wrong; more tests of the same kind would have caught none of these. What you want is a consistency audit: a check whose inputs are two different representations of one fact, and whose assertion is that they still match. The concrete version turns out to be pleasantly mechanical. For the client-versus-server class, we listed every constraint in the database migrations and every bound enforced in the client, then looked for rows that appeared on only one side. That took an afternoon and found four gaps — the worst being an age gate that the app enforced carefully, with its own dedicated test, and that the database did not enforce at all. Anyone with the public API key could create an underage account. Three separate public documents already promised that gate existed.

There is a lesson underneath the lesson. The four gaps were not randomly distributed. They were almost all in the oldest tables — written before the team had internalised that the server is the boundary. The newer columns had their constraints. So a consistency audit does not just find bugs; it dates them. It shows you which of your beliefs were adopted after which parts of your system were built — which is a map of exactly where to look next.