Comparing an agent’s context without copying the expected answer

On 30 October 2026, a gate comparing a TypeScript port of a coding agent with its Python original checked 1,358 recorded model requests, from 17 recorded sessions, and found no part that differed. The same report gave the number of those requests the port had rebuilt completely, without copying anything from the original, and that number was zero. Both figures are correct, and the first is only worth having because the second sits next to it.

Requests are an attractive place to compare two agents. Nearly everything a harness does ends up in the next body it sends to the model: the system prompt, the tool definitions, the opening context, the state blocks describing the session and its sandbox, the conversation history, notices from background work, cache markers and parameters. If a port assembles the same body from the same inputs, then at that call the model has no way to tell which agent it is talking to. That is a strong statement, and you can test it without calling a model. It is also easy to fake by accident, because the recorded request you are comparing against contains every value the builder could want.

A transplant carries the answer with it

The idea began as a transplant. While directing the port I asked for context from recorded sessions of the original to be moved across, so that the port would pick up from the same state and we could see whether it did the same thing. I expected a few differences, such as a workspace path that isn’t the same on the new platform, and I wanted those treated as “known things that we willingly and knowingly and intentionally replace”, with everything else identical. A few minutes later I added the rule that shaped the rest of the work: “Use deterministic to check anything that could be byte identical,” keeping a model’s judgement for whatever couldn’t be.

The trouble with a transplant is that the expected answer travels in the same package as the input. Suppose the builder needs the block describing the sandbox, and the easiest place to find the values is the recorded request. You lift them out, hand them to production code and compare what it renders with the block you lifted them from. Each step looks like testing, and the result is a pass that could not have been anything else. A broad search-and-replace has a milder version of the same problem, since a rule general enough to reconcile paths can reconcile other things too. Nobody has to be careless for this to happen. It only takes a convenient value in the wrong place.

Two inputs, kept apart

The request gate keeps them apart. It runs no agent, calls no model and executes no tool. For each recorded core request it holds the expected body, which is what the original actually sent, and separately a set of session inputs captured from the recording: the state the agent read, the replies it had received, the tool results it was given and the notices in its inbox. Only the inputs go to the port’s production builder. The expected body is what the result gets compared against, and the two places it is used for anything else, which come up below, are kept out of the passes. The report states its own scope, “No inference, tool execution, broad string substitutions, or expected-to-builder projection”, and keeps digests of the expected bodies and the inputs for each recording, so you can tell what was compared. That separation reduces the risk of circular evidence without abolishing it, since a captured input is only as independent as its capture, but it moves the risk somewhere you can look at it.

The substitutions I had expected turned out not to be needed at this layer. The inputs come from the recording, so any path, ID or clock in a rebuilt part is the recording’s own, and the comparison can be structural JSON equality with no timing, path or content substitutions at all. The one concession is to storage. Recorded bodies are what the inference gateway persisted, so the comparison follows that gateway’s pinned schema storage contract, which preserves schema property order and its rules for empty message fields.

Every part of every request then ends in one of four states, so the table accounts for every request. Identical and differing mean what they say. Erroring covers a builder that throws or an expected body that isn’t available. Supplied means the recording doesn’t hold enough to compute the part, so it was copied from the original’s request. A supplied part is a hole in the evidence with a label on it, and it never counts as a pass. A request counts as complete only when every one of its parts was computed.

What zero and zero look like

This is part of the map from that run. No row contains a differing or erroring part.

The table describes two things at once. A computed row says something about the port’s code. A supplied row says the gate couldn’t compute that part from what it had, and it says nothing about the port in either direction. The regression gate passed with 70,978 computed checks and no failures, and it excluded 9,700 supplied checks from that figure. Conversation history is supplied in all 1,358 requests, because at the time it was being rebuilt by a separate tool that wasn’t yet part of the gate. That one row is enough to make zero differing requests coexist with zero complete ones.

Accounting by whole request is also too coarse in the other direction. A request counts as supplied in the tool-results row if any of its results is, so only 35 show as computed there. Across all 1,358 requests the gate also counts the pieces inside them: 55,810 tool-result atoms computed and identical, none differing, and 75,593 supplied where the recording lacks the bytes. Command results were mostly computed, 33,173 against 10,214, and file views mostly weren’t, 2,361 against 18,378. These numbers describe the same requests with different denominators, and neither can be quoted without its own.

Notices get a third treatment, because a notice is really two things: the decision to fire it at this point and the text it renders. Where the decision can only come from the original’s request, the gate takes it from there and lets the port’s renderer produce the text. In this run 46,934 came out identical. That says something about the templates and nothing about whether the port would have decided to fire them, so the figure is reported alongside the results and excluded from the passes.

Progress in this gate means turning supplied parts into computed ones, and the regression baseline is built to defend that. It tolerates unchanged known differences and improvements. It fails on new differences, renderer errors, changes to the corpus, zero coverage and lost independent coverage, and that last rule is the one I would copy first, because without it the cheapest way to turn a failing part green is to stop computing it. Each row of the table is therefore a piece of work with a name and a count, and a row can only move one way without someone noticing.

Three different questions

A result like this answers one question: given the inputs the original had at each call, does the port’s production code produce the same parts, wherever it can compute them? It leaves two others open. It doesn’t show that a running port would have arrived at those inputs, since the state, tool results and history are handed to it. That takes replaying a session through the running agent, which takes most of an hour for the corpus against a few minutes here, depends on timing, can stop early when its own kit has a bug, and has accepted normalisations of its own that shouldn’t be read into this gate’s stricter equality. This run also checked core requests only, so the evaluator, summary and overseer calls need their own computation.

The third question is what the model does next. My original message said the context should be identical, “And then have an identical outcome.” The first half of that can be tested. The second can’t, because two runs of one agent on one context differ when the model samples. Equal context takes the harness off the list of explanations for a different outcome, and that is the most it can do.

Chris Chabot, November 2026