Porting an agent by making omissions visible

When a model translates a module from one language to another, the result looks finished whether or not it is. If it drops the branch that handles an unusual provider error, the code on either side of the missing branch still compiles, because the model wrote that code as well, and nothing in the output marks the place where the branch used to be. On a single function you can catch this by reading the two versions side by side. In October 2026 I directed the port of a production coding agent, which my colleagues had built in Python, to TypeScript on our newer agent platform. At the revision we pinned, its source tree held about 340,000 lines in 721 files, counting blanks and comments, with roughly another 420,000 lines of tests beside it. Nobody reads that side by side, so we needed some other way of finding out what the translation had left out.

My requirement made the problem harder. The port had to come out byte-identical to the original in everything not specific to the new platform: context and request bytes, compaction, tools, tool discovery, prompts, the turn loop and the nudges. That rules out the usual comfort of a rewrite, in which a translation that captures the spirit of a module counts as a success, and it makes one missing sentence in a reminder a defect of the same kind as a missing function.

What the specification could and couldn’t do

Before any of the porting, we specified the agent’s behaviour as a set of state machines in Lean 4. Each subsystem has its own component, among them sessions, user input, the turn loop, prompt assembly, the transcript, compaction, inference, the tool framework, the terminal, sandboxes, source control, persistence and orchestration. A composed model steps them together, which is what allows a proof about a lease takeover, or about a stop that arrives during a command inside a tool call, over every reachable state. By 30 October the project held 9,864 theorem declarations in 52 modules. Of those, 1,218 are restatements collected in two index files, which leaves 8,646 proved in the modules themselves, and that day’s build finished with no theorem depending on sorryAx, the axiom Lean records when a proof has been left unfinished.

Some of these statements are a pleasure to have pinned down. A turn’s ending is recorded once, and a second proposal to end it changes nothing. The transcript only grows, apart from five named events that the original code itself defines. A fold never drops an entry it did not fold. After a session lease moves to a newer holder, a write carrying the old token is refused. This gave the engine its organisation, and it gave everyone working on it, people and agents alike, a precise statement of what each subsystem was supposed to guarantee.

It also has a hard boundary. The Lean model shows that the design is consistent and that its invariants hold. It does not show that either codebase implements that design, the Python included, and it was written against an earlier revision of the Python than the one the translation later pinned. It also holds no bytes. It can require that a compaction commit waits for its durable cut, but it knows nothing about the text of the prompt that produces the summary, or about whether a given helper made it across to TypeScript. When we inventoried just the original’s turn logic, it ran to 216 paths, each with a trigger, an exact injected text or constant, a placement in the request and a limit on how often it fires, and that material is a large part of what an agent harness is.

A checklist from the syntax tree

By the middle of October a TypeScript implementation built up in stages from that specification already existed, and it matched the original on most of a small set of scripted scenarios. Those scenarios proved one example of each family, a single 529 retry, a single unretryable 400, one steer queued before a tool cycle, and the inventory counted only 62 of its 216 paths as proved or registered as an allowed difference. What I needed was a way to find what the implementation didn’t have yet, and on 26 October I settled on a method that was deliberately unclever. Generate an exact checklist for each module of the Python, have a reviewer with a clean context compare the original with the port and list what is missing, port the gaps, and review again. The question I wanted answered was, “Did you actually implement everything, or is there anything you skipped?”

The checklist is what makes an omission countable. A short Python script takes a unit, either a file or a package, reads each file from git at the pinned revision, parses it with the standard ast module and walks the tree. It writes one unchecked line for every function and class, with its line range, and one for every string literal of at least 40 characters, with its line number and the opening of its text. A header states what each line promises: the item must exist in the port with the same behaviour, meaning its branches, constants, thresholds, exact model-facing strings, ordering and side effects.

The script understands nothing, and that is its value. It can’t skim, it can’t decide that a helper was unimportant, and since it reads only the original, nothing a porter does can shorten the list. It also has holes you should know about. A string under 40 characters isn’t listed, and neither is one that opens with a newline, as triple-quoted text often does. A file that fails to parse is skipped. A branch or a threshold has no line of its own and exists only as something the reviewer is told to check inside the function that holds it. So the checklist is a denominator for a review and not a description of semantics, which is all a script like this could honestly be.

The reviewer who wasn’t there

Each review is a new process. The pipeline shells out to a command-line coding agent, Codex or Claude Code, with a prompt, so a clean context isn’t a matter of discipline: the reviewer simply hasn’t seen the port being written. Its prompt opens, “You are a REVIEWER with a clean context. Do not edit any code.” It gets the pinned original, the port’s tree and the checklist, and it has to find the counterpart of every item and judge whether it is complete and identical in behaviour. Code tied to the old platform, such as its UI framework and file store, counts as done when the port reaches the same model-facing behaviour through the new platform’s interfaces.

The answer isn’t prose. The reviewer writes a JSON file with a done flag, a count of items checked and a list of gaps, and each gap names the checklist item, the Python file and line, the TypeScript file and line or the word MISSING, and what is wrong. done may only be true when that list is empty. A schema like this gives a reviewer nowhere to put a general impression, and it gives the next agent something it can act on without any of the reviewer’s context. That next agent is the porter, in another fresh process, told to close every gap, because “partial work is a failure”, with the same logic, branches, constants and exact strings and no redesign. It checks that the code typechecks, commits, and a new reviewer starts again.

The instruction against redesign is there because a capable translator sees things to improve. The turn-logic inventory lists defects in the original that a byte-identical port has to reproduce rather than fix, including a misspelled reminder, a classifier that looks for a status code as a bare substring, and retries on a couple of client errors. Fixing those is a separate decision from translating them, and a translator who fixes one silently looks exactly like a translator who lost it.

The inventory, the state of each unit and the stopping rule are all deterministic, while the translating and the judging are done by models and aren’t, so the loop is bounded and its outcomes are explicit. After three rounds with gaps remaining, a unit does not become done; its status says that gaps remain. A review file that can’t be parsed stops the unit as unreadable instead of passing by default. The reviews kept from before the final sprint show what the loop was worth. The last reviews of 71 units listed 289 gaps, with a median of two per unit and a maximum of 23, and 52 of those reviews said the unit wasn’t done. Those units had been through earlier rounds, and reviewers were still finding things.

The sprint, and what the ledger won’t say

A few days earlier I had said we needed to escape a trap of tiny steps and start making massive moves, and by the evening of 26 October the queue made the case for one. It held 293 units: 161 done, 31 in a section called hard residue, where gaps had survived three or more rounds, 22 started and 79 not yet reviewed. The method for the sprint that followed fits on a page. Each worker got a batch file of its own, so there was nothing to claim and nobody to wait for. For each unit it ported, started a fresh reviewer, closed gaps for at most three rounds, landed what was correct, noted what wasn’t and moved on. There was no typecheck, test or replay per unit, continuous integration was paused, one stitcher kept the main branch compiling, and the instructions said no questions: decide from the source and this page. Modules too large for one pass were split into parts with their own checklists and line ranges, followed by an integration review that listed the calls, constants and shared state crossing between the parts and checked each one. Between 26 and 28 October, 508 commits named 225 units. That measures the closing of a port already well under way, not the translation of 340,000 lines in three days.

When it was over, every one of the 293 units was either in the old done section or had at least one port commit, and it would have been easy, and not even false, to report 293 of 293. The ledger doesn’t. It reports 161 done beforehand and 132 with a port commit, and it says plainly that a port commit shows a porter closed the unit, not that a clean-context reviewer then found no gap. The sprint’s method required those reviews, but nothing records their outcome per unit. So “translated” has a number, “passed a final clean review” doesn’t, and anyone reading the ledger can see the difference.

Two other counts came at the port from different directions. A module checklist sees code from the inside, so a separate inventory listed what users and the model can actually reach: 108 application surfaces, 47 cards and elements in the conversation stream and 109 model-facing tools, all present or done by 30 October. The 109 tools were registered with identical descriptions and input schemas. Their results were harder, since only 29 had rows compared against recorded sessions, and for the other 80, reading both sources gave 39 at parity, 31 fixed and 10 known differences where the platform can’t reach the same outcome. The other count comes from the original’s test suite, where we translated 77 of 1,334 test files, about six percent. They pass, 1,649 tests with no failures, but whole modules such as orchestration have no translated file and a cited file doesn’t mean every case came across. The fraction matters, because 1,649 passing tests sounds like a great deal more than six percent.

A different claim

All of this makes particular omissions visible, and none of it says the port behaves like the original. A reviewer is still a reader, and reading can be wrong. Earlier that month a few rows of the turn-path inventory still rested on reading the Python, so they were settled by running it, and the set of statuses the original actually retried turned out to differ from what had been read: a 413 is never retried. What the checklist pipeline does is clear out the bulk of the differences cheaply, so that the expensive comparisons can spend their time on subtle problems instead of on a function nobody carried across. Those comparisons rebuild the requests the port would send and replay recorded sessions through it, and the first full round of replay, on the last day of the sprint, reported 8,607 differing rows across 16 recordings. One cause can produce hundreds of rows, so that figure says little about how many things were wrong. It does establish that “translated” and “behaves the same” are different claims.

Chris Chabot, November 2026