Alerts I can trust, because I broke them first

Why

An alert I do not trust is worse than no alert at all. If my phone buzzes for things that do not need me, I will learn to swipe it away without reading it — and then the one that actually matters gets swiped away too, by reflex, at which point I have paid for a monitoring system and bought myself a slower version of not being told. So the list of things allowed to interrupt me is deliberately tiny, everything else waits in a queue, and no alert path counts as working until I have broken the thing it watches and watched the message arrive.

The rule is: a silent phone means nothing needs me. Five events are allowed through — the assistant finished a job, the assistant needs an answer, a commit could not be signed, the box is heading for a crash, and the weekly budget is about to reset. On top of those there is an hourly digest of everything that went wrong, and one summary in the morning. That is the whole list.

I did not arrive at that list by being thoughtful. I arrived at it by getting it backwards and measuring the result. The first version of the policy tried to warn me before things needed attention. After one day of real use, the event I had explicitly said I did not want had fired twenty-eight times across seven sessions in two days. The two events I did want had fired zero times. The noisy one was a guess wearing a uniform: it asked “does this look like the risky situation?” instead of “did the risky situation actually happen?”, so it fired on text that merely mentioned a tool, and it blocked every unattended run outright, by demanding a human decision at an hour when nobody was there to make one. I deleted it, and moved the guarantee it was supposed to provide into the one piece of code that can observe the real failure directly, after the fact.

Everything else queues. Failures from scheduled jobs go into a spool, and one drainer — exactly one — empties it every hour and sends a single digest. Two drainers would race each other, and whichever ran first would take the alerts while the other reported an empty queue, splitting one incident across two notifications. When the queue is empty it sends absolutely nothing, so a healthy week is exactly as quiet as it was before I built any of this. The extra cadence is only paid on days something is actually wrong.

The nicest thing I have proved about this system, I could not prove from the box. The box can see what it sent; it cannot see what my phone actually displayed. So the check was screenshots of my own notification tray, matched line by line against what the box had archived: ten alerts across five notifications, every one of them the sanctioned hourly digest, nothing sneaking out of a scheduled job directly. That match is only good for the day it was taken — the renewable version of that check is another screenshot, and it is one of the few things here that structurally requires a human.

The other lesson came from a test. A notification test once escaped its sandbox and sent a real message to my phone, which was embarrassing and obvious. The quiet version was worse: a test that faked one output path but not a second one added later, so it wrote into the real state directory and polluted the exact log I was about to read as evidence. Nothing rang, so nothing announced it. A test fixture has to fake every path its subject writes through, not just the one that would be humiliating.

What broke

The memory alarm was completely silent during the exact event it exists for. The operating system's out-of-memory killer took down my long-running service one afternoon and permanently orphaned three attached sessions; the watchdog had run eighty-one seconds earlier and warned about nothing. It was right to be silent, which is the uncomfortable part. One of its triggers watched total memory against a ceiling the kill happened well underneath. The other watched a sixty-second average of system pressure — during an event that lasted thirty-five seconds. A sixty-second average cannot climb inside a thirty-five second event. That is arithmetic, not a badly chosen number, and the obvious fix of lowering the threshold would have bought me false alarms without touching the blind spot. It got a third trigger on a much shorter window and a faster check interval instead, and both original thresholds were left exactly where they were.