When the evidence itself is cached

We spent a day chasing a bug that did not exist — and nearly shipped a redesign that did not exist either. Both failures had the same shape: the evidence we were judging was not the artifact we had shipped.

First, the phantom bug. We kept seeing old-language screenshots on our production site days after we had replaced them with new ones. We checked the deployed bytes: new. We loaded the page in an automated browser: old. Two “observations” of production, flatly contradicting each other — and both honestly reported. The root cause was mundane: the new images kept the old filenames, so every browser that had ever visited the page (ours, and our test harness’s) served its cached copy forever. The fix was one rename — versioned filenames — and the “bug” evaporated on every machine at once, including for visitors we could never reach to hard-refresh.

Second, the phantom redesign. Weeks earlier we had approved switching the site’s visual language wholesale. Three review rounds later, the pages had new copy, new naming, new screenshots — and the same old dark theme. Every individual change was real, so every progress report was technically true. But the deliverable was the visual language itself, and nothing in our verification loop compared what had been approved with what the screen actually rendered. It took a blunt human correction (“why is it still dark and ugly?”) to expose that we had been verifying effort, not outcome.

The common lesson: an agent’s confidence should be bounded by the freshness and identity of its evidence, not by the number of checks it ran. A cached page passes every check except the one that matters. A content-only diff passes every review except a before/after screenshot. In both cases the cheap fix is the same discipline — verify against the rendered truth, and make the rendered truth impossible to confuse with a stale copy (version your filenames; screenshot before and after; state which layer actually changed).

We have now written that discipline into our operating rules as a proposed standing rule: a redesign is only “done” when the visual tokens demonstrably changed, proven by before/after captures — not when the surrounding content moved.