The alarm grew past the wire

Two days ago we fixed a small, embarrassing bug in a nightly audit. The audit runs against the real database, writes a human-readable report, and posts it to a chat channel — that post is the only way a person ever learns the result. The bug was that the post could fail and nothing would notice: the call didn’t treat a 4xx response as an error, and the “this run completed” timestamp was written before the send. So a run could finish, fail to reach anybody, and still look fresh.

We fixed both halves. The call now fails loudly. The completion stamp moved to aftera successful delivery, on the reasoning that from the reader’s point of view, “it didn’t run” and “it ran but you’ll never know” are the same event.

This morning a separate scheduled job — deliberately a different process, on a different schedule, because a monitor that shares a fate with the thing it monitors is decoration — asked the only question it is allowed to ask: when did the audit last finish? Answer: forty-seven hours ago. Two nights missing.

The audit had run both nights. It had run fine. The report was 3,660 characters one night and 4,340 the next, and the channel rejects any single message over two thousand. Both were refused whole. No stamp, correctly.

The interesting part isn’t the character limit. It’s the direction of the failure. Every report that had ever been delivered was under 1,400 characters. What changed wasn’t the limit — it was us. A few weeks ago we improved the report so each red line carries the underlying error text, because without it you can’t tell “the rule really broke” from “the test script lagged a legitimate change.” That was a good change. It also means report length grows with the number of problems. The channel’s capacity is fixed. The message’s size is a function of how bad the night was.

So the alarm is silent precisely on the nights it has the most to say. On a quiet night it works perfectly. You cannot find this bug by testing the happy path, and you cannot find it by watching production, because production looks fine on exactly the days you’re watching.

The generalisation we’re taking away, and adding to our pre-ship checklist: every alert channel has a capacity, and you have to measure your worst-case payload against it, not your typical one. Ask what the message looks like on the worst day you can construct — not the day you happen to be having. Anything that summarises failures is a candidate: length, attachment size, rate limits, retention.

The consolation is that the fix from two days ago paid for itself immediately. It didn’t prevent this outage. It converted a silent one into a loud one, forty-seven hours in, which is the entire job of that class of fix. We’d rather find out this way than the way we found the last one.