Separating an agent from its platform for differential testing
How to separate an agent from its platform with a mechanical test when verifying a port, and how to run the unmodified original as a deterministic oracle for differential testing.

On this page
When you verify a port or reimplementation, your comparison harness eventually gets sharp enough to surface every difference, and most of what it flags is not the system you are porting. Accounting for every discrepancy is the disciplined instinct, and for regressions it is the right one. Proving platform noise away never converges, while relaxing the bar abandons equivalence, so the system under test has to be bounded mechanically and the executed original used as the specification: differential testing against a reference implementation.
I hit this while porting our production coding agent, nearly 400,000 lines of Python, to TypeScript so it could run natively on my agent platform’s runtime with identical behavior. We verified the port by replaying recorded production sessions and comparing outbound model requests, and the comparison came to account for every one-second clock difference while the port still failed acceptance. Does your terminal say that it ran 6 seconds instead of 5 seconds? That is not the agent. Treating the agent and its host infrastructure as one system made the harness exact about platform variance while real divergences went unseen.
Defining the agent boundary
Deciding case by case whether a difference matters turns verification back into judgment, so the boundary has to be mechanical. A colleague had already moved the Python agent onto that same runtime behind a 7,400-line adapter; the other 98% ran unchanged, and nobody called it a different agent. That supplied the rule: for any line of the original, did the adapter have to change it? Unchanged code is agent behavior, including context assembly, steering, tool schemas, compaction, and error text, and must match exactly. Replaced lines are platform concerns like timers, identifiers, paths, and host mechanics, where the port follows its own runtime. In model-visible text, the template is agent behavior and the value is platform.
That line put nearly every remaining replay failure on the platform side, while checking the port in reverse exposed the larger risk. A fifth of the TypeScript codebase had no counterpart in the original, including most of a 25,000-line state machine derived from a formal model proved in Lean, a theorem prover. Whatever in that extra code reaches the model is behavior the original lacks, and no replay reveals it because the original never enters those states. The proofs hold, yet a specification of how an agent ought to work is not a port of how it does work.
Fail-safe normalization and one sweep
With the boundary explicit, one normalizer replaces per-difference exceptions, erasing an allowlist of measured platform values and failing safe: a rule that erases too little shows up as a difference, while constants and limits in prompts or tool definitions stay compared verbatim. A report missing a recording fails rather than reading zero.
Reconciling a codebase of this size in a change-test-change loop stalls behind replay runs after every edit. Instead, we mapped every change up front, made them all in one sweep, and tested once afterward. About 67,600 lines left the port, including most of the formal-model state machine. All 17 recordings then replayed with two differences in about 3,000 request bodies, running against stand-ins for the runtime, storage, and model rather than live infrastructure.
Running the original as an oracle
Session replays only cover paths recorded runs happened to take, leaving retry thresholds, refusal texts, parsing rules, and error branches untested. Translating by reading the original misses semantics that look routine until they diverge at runtime: Python’s str.strip() on an argument, int() on a settings string, or equality checks where a bool equals 1. Since the specification already runs, the reliable test executes the original Python and requires the TypeScript port to match its output on the same inputs.
Making an unmodified codebase deterministic enough to serve as an oracle puts every control outside its source. The harness installs the pinned Python original from its own lock file and verifies the source tree by digest. A clock shim loaded before the first import supplies scripted timestamps to wall-clock calls while monotonic timers and the event loop stay real. Every outbound effect, from HTTP calls to sandboxes and tool servers, routes to a recording stand-in that answers from a scenario script and aborts on any unscripted call. Each unit runs twice and must match byte for byte before saving a fixture, catching non-determinism before it becomes a false defect. The suite grew from 25 fixtures to about 200, spanning some 15,500 comparisons against the TypeScript port.
Holding later changes to parity
Parity against the executed original also binds later changes. Of 177 security fixes written for the port, 171 were checked against the original Python, while six were deferred until the original changes. Tightening validation or altering a refusal path the original still permits may improve the port in isolation, yet a port safer than its original is still a divergence the model sees. Applying those fixes to the original first and porting the delta keeps one source of truth instead of letting them drift.
Chris Chabot, October 2026