An automation stack built by AI agents, under tests

Why

The lab runs a pile of scheduled jobs I never look at: backups, audits, health checks, log writers. If one of them quietly stops working, the job will not tell me. I find out weeks later, when I go looking for something it was supposed to have been doing and it is not there. Almost all of this code is written by AI agents against a plan I wrote, which makes them fast, tireless and extremely confident — and confidence is the thing I cannot use as evidence. So the tests are not a nicety here. They are the entire reason I am allowed to stop paying attention.

Four rules do most of the work here, and every one of them is a scar.

A test that has never failed proves nothing. A green suite is not evidence that it is watching anything; it is evidence that it ran. So when I add one, I go and break the thing it is supposed to catch, confirm it goes red, and put it back. I do that several times per suite and report the result as a number — “six mutants, six killed”. If one of my deliberate breakages survives, that is not a lucky escape, it is a hole in the test, and the test is what gets fixed. This is the single practice that has caught the most rot, and more than one suite here has failed it on its first attempt.

Never judge coverage by name. For a while I checked whether a job was tested by searching for its filename across the test files. That measures nothing. One job matched because its name was a fragment of a different job’s name. Another matched because it appeared inside a list of expected filenames in an unrelated test. A third matched a test that covered its helper libraries and not the three-hundred-and-fifty-line program that used them. By that method two jobs looked uncovered; by actually tracing which tests execute which programs, six were. Writing the three missing tests immediately turned up two real defects, including a job that folded every possible failure into a success-shaped report, so a step that never ran still printed a cheerful summary.

Never keep a hand-written list of the tests. I had three such lists and all three had drifted. They are gone; the tests are discovered instead. The discovery step deliberately fails the build when it finds zero of them, because otherwise a typo in a search pattern reports success in an encouraging green colour, and that is exactly how nothing ran for two days under a passing badge.

Do the checking as early as you can afford to. Edits are checked at the keystroke: if an agent writes a shell script the linter objects to, the edit comes back blocked before it is ever saved. Certain shell commands are checked before they are allowed to run at all. Then the whole thing runs again in continuous integration, on a fresh clone, in a different timezone, with the linter pinned to an exact version — because my machine’s linter and the pinned one genuinely disagree about real code, so “it was clean on my laptop” is not evidence about anything.

Underneath all four is one habit that took me longest to learn: writing a defect down is not the same as fixing it. I once had the precise cause of a hang recorded, in writing, naming the exact line, five days before that hang bit me at the worst possible moment. The note was correct, specific, and completely useless, because nothing fails while a note is true. Now the standard is: land the fix, or file it somewhere that turns red on its own. A document is not a control.

What broke

A watchdog whose test fixtures described a world that had stopped existing. Its job was to tell me whether a nightly routine had actually run, and it did that by reading session transcripts. Then the routine moved to a different runner and the watchdog could not see it at all — while its own suite stayed green at thirty-nine of thirty-nine, because every fixture in it encoded the old world. A blind instrument with a perfect score, for weeks. I replaced it with one that asks the scheduler directly, and built its fixtures from real scheduler output captured before the script was written. The cost, stated plainly because I would rather write it down than rediscover it: a power cut in the middle of a run is now detected by nothing.