A test that has never failed proves nothing

Here is a rule I follow, and it is the rule that has caught the most real problems in my lab: before I trust a check, I break the thing it is checking and make sure it notices.

It sounds almost too obvious to say. A test exists to fail when something is wrong, so of course it fails when something is wrong. Except that nothing proves that until it has happened. A test that has only ever been green is indistinguishable from a test that cannot go red — one that checks the wrong file, or compares a value to itself, or quietly skips when its input is missing and reports the skip as a pass. From the outside, all of those look exactly like a healthy system. They are the same colour.

So the habit is small and mechanical. Take the thing the check guards. Break it in the most direct way you can: delete the line, flip the condition, plant the string the check is meant to catch. Run the check. It should go red. Then put the thing back and watch it go green again. If it stayed green while the thing was broken, the check is not a check; it is decoration, and it is worse than nothing, because it is telling you everything is fine.

The formal name for this is mutation testing: you make a small deliberate change — a mutant — and a good test “kills” it by failing. A mutant that survives is not a bug in the code; it is a hole in the test. I report the result as a plain count, so many mutants, so many killed, because “the tests pass” on its own tells you nothing about whether they could have failed.

The case that made this a rule for me was a watchdog. It was meant to notice when a scheduled job on one of my machines had stopped running, and it had a test suite, and the suite was green. It had been green for weeks. It was also wrong. The suite fed the watchdog a picture of how the machine was laid out — which jobs existed, where their records lived — and that picture had been true once. Then the machine changed. Jobs were renamed and moved. The watchdog, run for real, was looking in places that no longer existed, finding nothing, and treating “found nothing” as “nothing is wrong”. The tests could not see any of that, because the tests were still describing the old machine, where everything was fine.

Nothing in that suite was broken in the usual sense. Every assertion was correct about the world it described. The world had just moved on without it. That is the failure I now watch for most: not a test that is wrong, but a test that is right about something that is no longer true.

The fix had two halves. The first was to make the suite read the live layout of the machine instead of a frozen copy of it, so the tests and the real watchdog were looking at the same thing. The second was the mutation habit: break the job the watchdog guards, confirm the watchdog notices, put it back. If the machine changes again and the watchdog loses sight of it, that break stops being noticed, and the suite goes red instead of staying politely green.

Two things follow that are worth saying plainly.

First, a new check is not finished when it passes. It is finished when you have seen it fail for the right reason. That usually takes a couple of minutes, and it is the cheapest insurance I know of.

Second, the checks that matter most are the ones that go quiet when they cannot look. If a check could not reach the thing it measures, it must say “not measured” in a way that nobody could mistake for “clean”. Silence is the one result a check is never allowed to return, because silence is exactly what a broken check sounds like.