Replaying the deterministic part of an agent

A coding agent alternates between two kinds of work, and only one of them is random. The model returns text and tool calls that nobody can make it return twice. The harness then interprets that reply, dispatches the tools, updates its state, decides whether a nudge or a compaction is due and assembles the next request, and all of that is ordinary code that should do the same thing every time it is given the same reply. I put it this way in a message on 29 October 2026, about a week into using the idea on a port of a coding agent from Python to TypeScript: the harness “interprets tool calls, logic, and all that kind of jazz. It does stuff deterministically. That creates a new context, and that context gets sent to the model.” If you hold the model still, what is left to compare is that machinery.

Holding the model still takes a recording. For each session of the original agent, we kept the requests it sent to the model and the responses it got back, byte for byte as the inference gateway stored them, joined by request ID to the gateway’s logs, together with the agent’s runtime log, which gives the timing of passes, summaries, notices and compactions. In a replay the port’s real entry point runs from the first message to the last. Its production code does everything it normally does, with the platform services underneath stood in. The model isn’t the only thing outside the harness’s control, so a stand-in answers each model call with the recorded response for that point and each command from the recorded terminal and workspace. It routes calls by kind, so that a title request or an overseer review can’t consume a reply meant for the main loop, and it holds calls so that a steer arrives where it originally arrived. Since both agents now see the same answers, a difference in what the port sends, does or shows points at the port, unless the replay itself is at fault, and telling those two apart turned out to be a good part of the work.

What gets compared is wide on purpose. Every outbound request is checked part by part against the recording: system prompt, tools, state blocks, history, notices and cache markers. Every tool call, client event and count is compared as well, across core, evaluator, summary, overseer and keep-warm requests, cards and stream updates. The contract behind this names each surface and keeps views onto them, so that a report can say “the third block of the second message of call 7, at byte 412” instead of only that a body differs. It is just as explicit about what it leaves out: transports, checkpoint encodings, logs, telemetry, the platform’s own recovery flows, and what a real model would do with those bytes.

Six small sessions

I asked for recordings on 23 October, with the model, the effort and the rest stored alongside, and I asked for a small library first. I suspected there would be “100 little bugs in the core implementation”, and larger sessions would be premature until we had surfaced those as cheaply as we could. So the first six sessions cost $5.68 in total: a one-turn build, a steer delivered mid-turn followed by a second turn, exploring and editing an unfamiliar repository, a failing test run with its fix, a series of tool errors, and the agent stopping to ask a person a question.

Even sessions this small don’t do quite what you designed them for, which is why the manifest records both the behaviours a session was meant to exercise and those the recording actually shows. In the tool-errors session the agent wrapped each failing command so that it captured the exit status itself, so its terminal cards read success and the failures appear only in the tool results. In the question session it asked its question as the text of an ended turn and not through the dedicated card.

The first full run made 2,689 checks: 2,218 passed, 471 failed and 11 couldn’t be checked yet. Of the failures, 258 were attributed to the replay environment or to configuration, and the other 213 collapsed to about 40 patterns. The port lacked two tools the original offered. A sandbox tool ignored a GPU setting. One terminal card reported a size a byte larger than the original’s. The original sent a one-token call to keep the prompt cache warm after each session’s last turn, and the port didn’t. One pattern showed the reference itself had moved, since the live original was newer than the revision the port came from and its system prompt had changed. And one needed more thought before anyone called it a defect: after a steer, the provider refused a call because a replayed thinking block no longer matched its conversation, and the original retried at once and finished in 13 calls while the port finished in 9.

Bulldozing to the next difference

The working loop I kept asking for had a name, bulldozing: run it, hit a problem, fix the problem, restore from the checkpoint, do it again, “keep bouncing against that obstacle until you fixed it”, then go forward to the next one. Once a small session was clean, we would branch out into sessions that behave differently, with compaction, failing tool calls, bugs caught by the linter at the end of a turn, tool discovery and memories, and do the same exercise across them in parallel. By 30 October the corpus had 17 recordings, with sessions covering large inputs, repeated steers, background jobs, web and media work and orchestrated workers, and the largest ran to 275 core calls through three compactions.

That loop only works if a restart is cheap. On 28 October long replays were taking eight to twelve hours, and I asked for something more like fifteen minutes: “we started at step 406 because we know step 407 things are going to go wrong.” A replay can be frozen at a point and resumed from it, each run is capped at ten minutes and long sessions run in segments, so a fix can be checked from the step it affects. I had also asked whether workers could stop building their own replay environments for every job. Replays came to take one of a small number of slots on a shared host, four by default and one reserved for CI, with dependency caches shared and source trees and cassette files never shared for writing.

The comparator needed the same attention as the loop. A check fails at the first byte that differs and reports the offset and the text on each side, which gives you a place to start. It also means the rest of that body goes unread, and in one orchestration recording a cost cell and some sub-agent sections sat behind an earlier difference through several rounds. So a second tool compares every paired body to its end and lists every line that differs, and a body passes only when nothing is left or each remaining line matches a named rule.

Keeping the masks honest

Normalisation is where a replay can start telling you what you want to hear. The replay’s table of substitutions is closed: wall-clock instants, artifact names built from them, times of day and elapsed durations in tool headers. Nothing else is substituted. The masks the line-by-line comparison uses for IDs are narrower than you would first write them, so a hex string only counts as generated when it contains both letters and digits, because, as the comment in the tool says, “an eight-digit number or an eight-letter word is content, not a generated id.” Attributing a failure to a known cause only groups it, and the check still fails and still sets the exit status. A difference isn’t excused because it might follow from an earlier one, and a missing or extra request stays a failure.

Even a necessary normalisation cuts both ways. Recordings hold bodies as the gateway stored them, and the gateway drops some schema fields and sorts top-level keys, so the port’s raw bodies are re-encoded through the gateway’s own types first, because otherwise the comparison would report differences neither agent made. Reading both sides in that encoding also hid a stream member on the port’s requests until a separate check started looking at the raw top-level members.

Five kinds of difference

Every difference falls into exactly one class, and the class decides what happens to it.

• Behaviour. The port sends, does or shows something different. It is a defect, fixed in the port with a test that fails first.
• Harness fidelity. The replay kit paces or serves something differently from the original run, in a way the agent can see. It is a defect in the kit, fixed there.
• Documented timing. The value depends on the clock, such as an age in seconds. It is accepted only when the original’s own log proves the cause.
• Recording gap. The recording doesn’t hold the input that produced the original’s value. It is accepted with proof of absence, and it remains missing evidence rather than a match.
• Platform difference. The agent is faithfully reporting a fact of its platform, such as a workspace’s file count. It is accepted only by an explicit ruling from me.

Two rules keep this honest. A count of requests, tool calls, messages or events is always behaviour, never timing. And nothing is accepted as timing on suspicion.

Verification ran in rounds: replay the recordings on a fixed pair of commits, classify every difference, give each open cause an owner, and let the next round confirm the fixes. Seven rounds ran between 28 and 30 October. The totals of differing rows, in the rounds that produced one, read 8,607, 5,352, 8,021, 6,401 and 4,312. That isn’t a progress bar, because one cause can produce hundreds of rows. In round seven one compaction recording had 912 differing rows and a single open model-facing row, while a larger multi-turn app had 2,573, among them six state-block cells off by one or two seconds that look exactly like timing and stayed open because nobody could yet prove they were.

That round left two of the 16 recordings identical and nine more with only documented differences or none the model could see. The other five show the classes doing their work: the compaction recording, later closed as a recording gap, which stays missing evidence; the multi-turn app with those unproved cells; one with a cause fixed after the round and awaiting the next; and two orchestration sessions that stopped early on a regression in the replay kit itself, which says nothing yet about the port. The long session, run separately, shows how small a difference these rules keep alive: its compaction, evaluator and overseer request counts all matched, while summaries came to 245 against 246, and a count that differs by one is still behaviour.

The live run doesn’t get the last word

Replay says nothing about behaviour no recording exercised, about real provider availability, about throughput or about every client journey. Live runs reach some of that, and they are a separate layer. On 25 October both agents got the same brief, a production job-queue service, and the same steers at the same offsets, with real models. The port finished with 764 tests passing at about 98.5 percent branch coverage, and the original reported 776 at about 98.9 percent, each for around $50. The two runs produced similar results. The same session then found a real defect: a fifth steer pushed both past the compaction threshold, and the port’s compaction stalled on a 90-second client deadline and an endless retry. The deadline and the retry were both fixed in the next release.

That is what a live run is good for. Model sampling makes two runs of one agent differ, so it can find problems and can’t certify their absence, and an equivalent result doesn’t outrank a difference found deterministically. A one-token request that keeps a cache warm after the last turn of a session has no effect on whether the job queue gets built, and no test count would show it. It turned up because a recording said the original had sent a request the port had not.

Chris Chabot, November 2026