The fixture that couldn’t tell them apart

Yesterday an agent shipped a small behavioural change: an identity switcher should list the currently active identity first, instead of always pinning the account identity to the top. The agent did everything our discipline asks for — wrote the test first, watched it fail, made it pass, ran the full gate. Green across the board. And the first version of that test was worthless.

In the default test fixture, the active identity wasthe account identity. Under the old rule (account first) and the new rule (current first), the expected list order came out identical. The test asserted an ordering that both implementations produce. It went red before the fix only because of an unrelated setup detail, and it would have stayed green if someone later reverted the feature. A test like that isn’t evidence. It’s a decoration that happens to be the right colour.

The fix was to ask one question before writing the assertion: what is the smallest input on which the old rule and the new rule disagree? For an ordering rule, that means constructing the discriminating state — make the active identity something other than the account — and asserting the order that only the new rule produces. Once the fixture could tell the two rules apart, the red test finally meant “the feature is missing” and the green test finally meant “the feature is there.”

This generalizes past orderings. Any change of the form replace rule A with rule B has a truth table, and your fixture is one row of it. If A and B agree on that row, the red-then-green ceremony proves nothing about the change — it proves your setup code runs. Defaults are where this bites hardest, because default fixtures are engineered to be unremarkable, and unremarkable inputs are exactly where competing rules tend to coincide.

So “find the discriminating input first” is now a required step for any rule-replacement work: name the input where old and new diverge, pin the test to it, and if you can’t find one, that isn’t a testing inconvenience — that’s the discovery that your change has no observable behaviour.

The embarrassing part is how familiar this is. Property-based testing people have said it for years: a test is only as strong as the distribution of inputs it visits. We rediscovered it the slow way, on a list with four avatars in it.