Loops fail at their exits

I fixed two separate faults in xNAUT on the same day. Each fix was correct. Each had a test. Together they produced a retry loop that could never stop, because the brake was counting evidence that the improved code no longer produced.

2 September 2026 · André · Part 1 of 3

The clerk and the receipts

xNAUT has a component called the sweep. Picture a clerk who walks the office every three minutes, looks for work someone has marked finished, and checks it. Nobody has to ask. That is the entire point. It keeps working the board after everyone has gone home.

Checking costs time and money, so the clerk must not check the same thing forever. It uses a rule anyone would recognise. Three strikes. Fail three times in a row and stop trying, and say so out loud.

Now the one detail that matters. The clerk counts strikes by counting receipts. Every check that runs leaves a record saying whether it passed or failed. Three failure records means give up. The records are the memory. There is no other tally anywhere.

Fix one: stop the silent typo

A configuration file names where a check should run. Two names are valid. I wrote a third that does not exist.

Nothing complained. The code quietly substituted one of the real options and carried on, so the check ran in the wrong place and failed for a reason unrelated to the actual problem. A misleading answer is worse than no answer, so I made an unrecognised name a hard refusal, by name, with the valid options listed. This is the right fix and I would make it again.

Fix two: stop the endless retrying

Separately, the clerk had no brake at all. A failed check was tried again next round, and the next. In one afternoon it produced 134 identical failure records across two items, every one failing for the same permanent reason, over six hours and forty-two minutes.

So I added the three-strikes rule. It worked, and I watched it work: three failures at exactly thirty-minute spacing, then a single line saying it was giving up, then silence.

Then the two fixes met

The whole bug fits in one sentence.

Making the error message better made the error invisible to the thing meant to stop it. The clerk kept trying forever, not because three strikes was broken, but because no strikes were ever recorded. I measured nine identical failures in twenty-seven minutes with no brake engaging, then four more on the next build to confirm it was not a fluke.

Nothing was lost or corrupted. No code, no customer data, no work. This is a waste and noise fault rather than a destructive one. The cost is slow. Every useless retry burns a little compute, and every junk entry buries the real entry beside it. A log that is mostly noise is a log nobody reads, and then a genuine failure sits in it unnoticed for a day.

The same loop, failing the other way

Two days later the same function broke in the opposite direction, which is the part that convinced me this was a pattern rather than an anecdote.

The clerk asks for the oldest finished item and checks it. Once three strikes existed, an item could be held. But the code asked for the oldest item and stopped if that one was held, instead of moving to the next. So a single given-up item silently halted the entire board.

I measured it. A clean item, no failures, nothing that could hold it, drew nothing for twenty-five minutes across eight rounds while a held item sat ahead of it in the queue. Park the held one, and the clean one was picked up on the very next round. Another item was starved the same way for over two hours.

The whole fault is one word. This is the real change, trimmed of the parts that do not matter:

// before: ask for the oldest item, and give up if that one is held
if let Some(item) = oldest_awaiting_review(&items) {
    if held(item) { return }   // stops the entire board
    verify(item)
}

// after: walk them, and step over the ones that are held
for item in awaiting_review(&items) {
    if held(item) { continue } // skip this one, keep looking
    verify(item);
    return
}

One loop. One week. It failed by never exiting, and it failed by exiting too early. Both times the loop body was correct.

Why it survives review

Both fixes have tests. Both sets pass. Neither test is lazy or wrong. You could review either change on its own, carefully, with full knowledge of the system, and approve it correctly.

That is because the fault is not in either change. It is in the relationship between them, and a relationship has no owner. No file contains it. No test naturally covers it. Each author looks at their own piece and each piece is fine.

That reframes the whole problem. The dangerous edit is not the one that changes a loop. It is the one that changes what the loop is counting, in a different file, for a good reason, with a passing test.

You cannot see it, and that is the danger

Everything about this bug looks healthy from the outside, and that is not bad luck. It is what the failure mode is made of.

The loop looks fine. Read it and you see a correct brake with a sensible threshold. Each fix looks fine. Review either diff and you approve it, rightly. The tests pass, and they are not bad tests; they check what they were written to check, correctly.

The logs look fine too, which is the part that surprised me. This is a slow loop. One round every three minutes. It does not spike a CPU or fill a disk. In a log it reads exactly like a healthy heartbeat, which is what a periodic checker is supposed to look like. My first version of this fault ran for six hours and forty-two minutes and looked, at a glance, like a system doing its job.

That is the sentence I keep coming back to. You cannot tell a working brake from an absent one by watching, because a working brake is quiet and a broken brake is quiet. The only way to know is to make it stop something, on purpose, and watch it happen. Until you have seen a loop exit, you do not know that it can.

What I changed, and what I ask now

A refusal leaves the same paperwork as a failure

This is the actual repair. If the system declines to run a check, it still writes a record saying so, carrying the reason. The brake can count it and a human reading the drawer can see why. One rule for every reason a check might not run, rather than a patch for the one I happened to hit.

A held item is skipped, not treated as the end of the queue

Candidates are walked oldest first and a held one is stepped over. Still one check per round; the limit was never the problem.

I watch the brake, not just the engine

Because silence proves nothing, a brake now has to be seen engaging. Mine was seen doing it. Three failures, one line saying it was giving up, then quiet. One observed exit is worth more than a month of the loop looking well behaved.

The question I ask in review now

What produces the thing this loop stops on, and who else can change it? Ten seconds. It would have caught this before it shipped, and it is the only part of this post I would ask anyone else to copy.

The uncomfortable part

I wrote a commit message saying this was fixed, and it was not. The fix was merged, pushed, and deployed to a second Mac that exists to test it, where it was caught by the process described in Part 3. It never reached a release, and that is the only reason this is a story about process rather than an incident. The order matters: the rig is not a formality between me and users, it is the thing standing where users would otherwise be.

The rest of the week was no tidier. Two further faults were mine, one of which took the rig down for an hour. The record count in my own first write-up was wrong three times over. I said 134 records for one item in about an hour, when it was 106 plus 28 across two items over six hours and forty-two minutes. At one record per three-minute round, 134 in an hour was never arithmetically possible, and nobody noticed until a machine counted.

I am telling it this way because the alternative is a tidy post about how careful I am, written by the person who built the infinite loop and then signed off on it.