Directing AI to build, and how you know it worked

Most of what runs my home lab was written by an AI. I directed it: I said what I wanted, read what came back, and decided whether to keep it. That sounds like the easy half of the job. It turned out to be the whole job, and the part that takes skill is not the asking. It is knowing whether the thing you got back actually works.

The trap is simple. An AI reports a mistake in exactly the same tone as a success. “Done — all tests pass” reads the same whether the tests ran, whether they were the right tests, or whether they exist at all. A person who is unsure usually sounds unsure; a model does not. So if I judge the work by how convincing the summary sounds, I am judging the wrong thing, and I will be wrong at the worst possible moment — on the change I did not look at closely because the report was so tidy.

The fix is to move the judgement earlier. Before any work starts, I write down what “finished” will be checked against. Not “make the backup better”, but “a restore from last night’s backup produces these files, and this check fails if one is missing”. Once that exists, the question at the end is no longer “does this look right?” — which I am bad at answering for code I did not write — but “did the check pass?”, which a machine can answer for me, every time, without getting tired or charmed.

That changes what a test suite is for. It stops being a demonstration that the code works and becomes the acceptance: the written contract the work is measured against. I will only accept a worker’s output on verification, and the worker being an AI does not change that; it makes it matter more, because an AI can produce far more plausible-looking work in an afternoon than I can review by reading.

It also changes who owns what. The AI can write the code, and it can even write the tests, but the acceptance criteria are mine. If I let the same system write both the work and the test that grades the work, I have built something that marks its own homework. The useful split is that I decide what must be true, the AI does the building, and a check that neither of us can talk our way past decides whether we are finished.

A few habits have come out of that.

Done means landed. A change is not done when it is written, or when it passes on the machine it was written on, or when a pull request is green and waiting. It is done when it is merged, deployed, and measured from outside — the page loads from the real address, the scheduled job ran on its own and left evidence. Work that is built but not landed has a way of piling up quietly, and every piece of it is something I believe is finished that is not.

Measured beats remembered. When a report says the build is green, I ask for the command that showed it, not the claim. A figure from memory, or a colour from yesterday, is a pointer to a measurement, not a measurement. More than once the honest answer to “is it working?” has been “I have not checked since Tuesday”, and that is a perfectly good answer as long as it is the one given.

“Not measured” is not “clean”. If a check could not run — the machine was unreachable, a tool was missing — it has to say so in a way that cannot be mistaken for a pass. A check that goes quiet when it cannot look is worse than no check, because silence reads as good news.

One reviewer who was not in the room. When something matters, I have the work read by a separate pass that sees the code and not the author’s explanation of it. An explanation is persuasive by nature; a fresh read of the diff is not.

None of this is specific to AI. It is what good engineering teams already do with human contributors. What is different is the volume. An AI can hand me more work in a day than I could check by eye in a week, so the checking has to be built into the environment rather than done by hand. In that sense the most important thing in my lab is not any one service. It is the set of checks that lets me hand work off and still know, without taking anybody’s word for it, whether it is finished.