Verifying a coding agent port without a model in the loop

How to verify a ported agent harness one step at a time with no model in the loop, by comparing each next request byte for byte and never counting supplied inputs as passes.

October 20264 min readPortingVerificationAgent systemsPorting a production coding agent

Roughly parallel graphite and dry red strokes fill the canvas left to right, with bare bone gaps, some ringed with pale blue scrapes.

Proving that a ported coding agent behaves identically to the original starts from an awkward fact: two runs of the original rarely finish the same way. Live side-by-side runs find gross defects early, though model sampling prevents them from certifying that subtler divergence is absent. Clean-context LLM reviewers help during translation, yet their verdicts vary between runs, every pass costs tokens to re-read both codebases, and a clean report says nothing about what the reviewer skipped. Full-session replay, feeding recorded model replies into the port from turn zero, gives stronger evidence, except a long session takes eight to twelve hours and the first mismatch cascades through every subsequent turn.

I hit all three limits while porting a production coding agent that my colleagues built and my company ships, about 340,000 lines of Python, to TypeScript on a newer agent platform, where our last 73 clean-context reviews still listed 289 gaps. I settled on differential testing at the right boundary. An agent looks nondeterministic from the outside, yet the harness is a deterministic state machine wrapped around one stochastic call: it parses the model’s reply, runs tool logic and state updates, and builds the next request. Because our inference service stores every request and response body, adjacent logged turns already hold both ends of that transition: the reply that left the model and the next request the harness built from it. Checking that the port produces byte-identical requests across every recorded step turns equivalence over all visited transitions into an equality check, with no model in the loop.

Testing one transition at a time

Starting from the Python agent’s recorded state at step N, the check feeds the recorded reply, raw tool outputs, and queued notices through the TypeScript port to build request N+1: system prompt, tool definitions, state blocks, tool results, file views, notices, conversation history, cache markers, and model parameters. Neither the model nor the sandbox executes; their recorded outputs serve as inputs. That payload is compared byte for byte against the Python agent’s request, after a closed list of substitutions for generated IDs, timestamps, and paths.

Because every step loads the original’s recorded state directly, mismatches never cascade across turns, and any failure identifies the exact step and request component that diverged. The check also has to call production code, not a test copy: formatting and rendering logic that lived inline in the port’s runtime was refactored into exported functions shared by the runtime and the check. Timing-dependent behavior, such as when background helpers trigger, stays outside single-step assembly and remains with a smaller set of full replays.

Why supplied inputs cannot count as passes

Recorded sessions rarely retain every intermediate variable needed to recompute a request from raw inputs. Rendering a file view, for instance, requires knowing which revision is open, which changes with earlier writes including shell commands that mutate files without naming them. If the check quietly copies missing fields from the expected request to fill the gap, it becomes remarkably reliable at passing. An earlier version failed in exactly that way: it stayed green while full replays kept turning up differences, because tool results, notices, history, state blocks, and cache placement were copied from the recording and passed by construction.

Every part of a request therefore gets one of three outcomes: identical, differing, or supplied. Any part copied from the original because it cannot be derived from recorded tool outputs and the agent’s runtime logs stays marked supplied, with an explicit reason, and never counts as a pass.

With that rule in place, progress became converting supplied parts into computed ones, deriving one family of parts at a time from raw inputs and fixing each mismatch that surfaced. In fifteen hours on our recorded corpus of 1,358 requests, the computed share of tool-result parts climbed from about 42% to about 70%, and roughly 98,000 computed checks passed with no failures in about three minutes, against forty minutes for a full replay round.

Widening the check to production sessions

Per-step equality only proves equivalence over the transitions in the corpus, and purpose-recorded sessions cover scripted flows more than the edge cases engineers hit in daily work. We already had a wider corpus sitting in storage: colleagues’ past production sessions, reconstructed from the inference service’s stored bodies and the agent’s runtime logs.

We ran the check across 21 of those sessions, totaling about 9,300 requests, plus a 55,000-call session kept around as a “final boss,” though storage permissions left only about 500 of its requests with readable bodies, and only those were checked. None of those requests could be certified identical: they lacked workspace snapshots, so parts built from stand-ins stayed supplied, and they ran mostly newer builds of the Python agent than the pinned revision we were porting. Even so, every difference was assigned a single cause: port defect, reconstruction gap, missing capture data, or version drift, with version drift accepted only when traced to a dated upstream commit showing the session ran a different build. Across the entire set, no port defect turned up.

Chris Chabot, October 2026