# Reviewing code that won’t hold still Author: Chris Chabot Canonical URL: https://chabot.dev/blog/reviewing-code-that-wont-hold-still/ Published: 2026-11-03 Binding review approvals and test results to the exact source revision they examined, and stopping when the latest change has gone unread. If an agent has to stop and wait every time a reviewer reads its diff, the review costs you all the work the agent isn’t doing in the meantime. So you let it carry on, and then the review and the code start to drift apart. The agent fixes the typo the reviewer is about to complain about, rewrites the function the reviewer is about to approve, runs the formatter over the whole file. A complaint that arrives late is a nuisance. An approval that arrives late is a real problem, because if it attaches itself to whatever the tree holds at that moment, it certifies code the reviewer never read. Late in August 2026 I added overlapping review to the agent host I was working on. The first design question was what exactly an approval approves. The answer was a pair: a source mutation revision, which the host advances as the source changes, and a digest of the diff, synthesized from the host’s own ledger of file changes. The code is emphatic that this is never Git HEAD. An agent edits its working tree long before it commits anything, so HEAD can sit still through a whole session of changes, and what it names may have little to do with what the reviewer read. Counting changes made through the file tools is the easy part. A shell command might have changed anything or nothing, so the revision tracker also notes actions that were capable of mutating source and actions it couldn’t classify, carries a tainted flag, and has an explicit step for re-establishing its authority. I like that it has a way of admitting it no longer knows. A revision number also assumes that writes happen in a defined order. When a model response contains several tool calls, consecutive reads overlap up to a limit, each mutation runs alone after everything before it has settled, results come back in source order, and an abort stops later segments from starting, so a read can’t race ahead of a write requested before it. ## Scheduling around a moving target With the identity settled, the coordinator only has to schedule. The agent queues a review and keeps working. At most one provider round runs at a time, and revisions that appear while it runs coalesce down to the newest, so a burst of edits costs one further round rather than one per edit. When a result comes back it’s checked against the current revision and the request epoch, and if either has moved the result is dropped, approvals and findings alike. Dropping the stale approval is the purpose of the whole exercise. Dropping stale findings is less obvious, since a complaint about the previous revision may well still hold, but one that’s still true should reappear in the round that reads the newest code, and one that has since been fixed would only send the agent after a problem it has already solved. The epochs cover what a revision number can’t. When the user changes the request, the request epoch moves and everything reviewed or approved so far is cleared, because code approved against the old goal hasn’t been approved against the new one. Turns are fenced the same way, so a reviewer that answers after its turn was aborted has nothing left to deliver to. Approval is silent, since there’s no reason to interrupt an agent to tell it nothing is wrong. A current major or blocker finding is delivered exactly once, steered into the run if it’s still active or sent as an explicit remediation prompt once the main body of work has settled, and at most one low-effort rereview follows. Explicit reviews, automatic reviews and rereviews all draw on one shared budget of provider rounds. ## Deciding when to stop The end of the turn is where the design has to commit, because an agent and a reviewer could otherwise chase each other indefinitely. The host settles in a fixed order: drain the round in progress, flush the current revision with a bounded wait, reviewing it against the user’s persisted goal if nothing has yet, deliver any remediation, settle running jobs, allow the one rereview and record the outcome. A wait that times out leaves the review unavailable and incomplete. Exhausting the round budget ends in the same place. The host then compares snapshots of the source from before and after settlement, and if something changed in that last window the receipt says so: the review is incomplete because the source changed after final settlement. At that point there are two honest options, to keep reviewing until the source stops changing, with no guarantee that it ever will, or to stop and record that the last change went unread. I took the second. In this overlapping mode the reviewer is advisory, and an incomplete review doesn’t stop the turn from ending. What an incomplete review can’t do is turn into an approval. A current approval records that a model read this diff and raised no major finding. ## The same problem with test results Test results go stale in the same way, and they have a way of lying besides. Agents routinely summarize a build or test run through a pipe: ```sh 2>&1 | tail -30 ``` The shell reports the exit status of a pipeline’s last stage, which here belongs to the summarizer, and a summarizer essentially always succeeds. In one run a hard test failure was reported to the model as exit 0 three separate times, and the end-of-turn verification gate consumed those receipts as green evidence of a passing suite. Nobody in that chain did anything unusual: `tail` did its job, and the gate believed the number it was given. Bash records every stage’s status the whole time in `PIPESTATUS`, so the repair was for the execution layer to read the underlying statuses and keep them apart from presentation, including cases where a display pipe closes early and the producer dies of SIGPIPE. A hidden failure is only one way a green result can mislead. A test selector that matches nothing exits zero having tested nothing, and an honest pass goes out of date the moment the agent makes one more edit. So the record I built for the end of a turn counts a verification as evidence only when several things hold at once: a classifier recognizes it as a build, test, lint, typecheck or named check, it exits zero, its pipeline status is honest, it selected a nonzero amount of work, its scope covers the change, and no mutation has happened since, because recording a mutation invalidates the receipt. The rest of the turn is treated just as conservatively. Pending jobs, child notices, unresolved findings, failed tool calls, an abnormal ending or anything simply unknown all leave the evidence incomplete. Recognizing the command can tell us that it invoked a test runner; deciding whether it was the right test still requires judgment. The collector also needs to receive the events it is meant to judge. The host was feeding this record its jobs, review state, child notices and abnormal endings, but nothing outside the tests called the two methods that record mutations and verifications. The tests exercised the rules; they did not establish that a real command had reached the collector. That connection needs its own check: run a command, observe its receipt, then verify that a subsequent edit invalidates it.