Winning on average, benched for variance

We ran a real out-of-sample test on a forecasting model this week, and it passed. Pooled across the whole test window, it beat the naive baseline on both metrics we care about — it called direction right more often, and its rankings correlated positively with what actually happened, where the baseline’s correlation was negative. On the strength of those two numbers, the obvious move is to wire it into the daily pipeline. We didn’t. It’s running in the shadows with zero say over anything, and the reason is worth writing down.

The pooled numbers are an average, and an average is a summary that hides its own variance. When we broke the same test out day by day, the model won bothaxes on only half the days. The other half it won one and lost the other, or lost both. Worse, the spread of its per-instrument forecasts — the gap between its confident-up and its confident-down calls — straddled zero on most cases. Translated: on a given day, for a given instrument, the model’s signal was about as likely to point the wrong way as the right way, and the aggregate edge came from the days it happened to be right being right by a bit more than the days it was wrong were wrong. That is a real edge. It is not a dependable one.

The distinction matters because production authority is a daily grant, not an average one. A model that’s right on average but wrong half the days doesn’t get to touch a decision that has to be made every day — because the decision doesn’t consume the average, it consumes today’s call. A pooled metric answers “over a long enough window, is this better than nothing?” The daily pipeline asks a harder question: “is thiscall, today, one I’d stake a position on?” Those are different bars, and the second is the one that binds.

So the model got a conditional yes: admitted as a tagged, inferred, secondary overlay — visible, logged, and completely powerless. Zero weight on any score, no place on the daily schedule, no downstream consumer. It has to earn promotion the slow way: a live window long enough that per-day consistency, not pooled luck, is what’s showing through.

The rule we’re keeping: a promotion gate reads the distribution, not the mean. Before a model earns a vote, look at how often it wins on the axis that matters, measured on the unit of time the decision is actually made in, and how much of its signal is indistinguishable from noise. An edge that only exists after you average away the bad days is an edge you can’t spend on any particular day — which is the only kind of day you ever get.