Correcting the grader without rewriting the run

By the end of my 31 August 2026 evaluation, coding agents had put eight apps on Ferrite, the application platform I own, and all eight had passed an independent browser check. Every word of that is true, and it would have been a poor report of what happened, which was five passes from eight fresh attempts, three separate repairs, and one repair session that never completed. Keeping those results apart was harder than it sounds, because the grader itself turned out to be wrong, and correcting a grader is the moment when a benchmark’s history is most likely to get rewritten.

The harness, which I led with agents among its contributors, gave real clients the job of building and deploying small apps through Ferrite’s MCP tools: Codex with GPT-5.6 Terra, Claude Code with Sonnet 5, and Pi with both DeepSeek V4 Flash and GLM 5.3 Flash. The examples below come from a small paint calculator that works out how many tins a room needs. Each run left a structured record with separate layers for source artifacts, admission, deployment, probes and browser behaviour, and a layer that was never reached was recorded as absent rather than failed. A judge could add colour to a record but could never flip a deterministic red.

The first automatic count was four passes out of eight, and it was wrong in both directions. Two of the Pi runs, one on each model, had failed because the parser didn’t recognise Pi’s tool names. It looked for the prefix mcp__ferrite__, Pi’s direct names have a single underscore at that point, and so the records showed zero MCP calls for sessions whose transcripts plainly showed a deployment. As far as the grader could tell, those agents had never touched the platform. The GLM run had also tripped the browser check’s list of failure markers by displaying “Please check: room length”, which is a calculator asking for valid input. Neither verdict was overturned on anybody’s opinion: a corrected parser re-read the same transcripts, an independent browser journey ran against the same apps, and both passed without a line of either app changing. The third change went the other way. One of Codex’s apps had correct arithmetic, but a stricter mobile recheck found 519 pixels of content in a 390-pixel viewport, so it moved from pass to fail. Two up and one down takes four to five.

That downward move is the one I’d look at hardest. Anyone correcting a grader has an interest in the higher score, and a round of corrections that only ever helps the agents deserves some suspicion. No agent ran again for any of these three changes and no app was touched. What changed was what the grader could see.

The mechanics matter more than the arithmetic. Original run records were never edited. A separate assembly script joined each record to any later browser check, stored SHA-256 hashes of the record and transcript, and wrote an observation in which the original automatic verdict and original MCP count sat beside the corrected values. A correction cited the sequence numbers of the transcript events that justified it, so a reader could go and check the claim rather than trust the corrector. The tool-name fix stayed narrow as well: /^mcp__ferrite_(?!_)(?!.*__)/ accepts a single underscore and rejects any name with a later double underscore, so a different server whose name merely starts the same way doesn’t get counted. The zeros are still in the original records, and anyone who wants to know how wrong the first grader was can still find out.

Repairs are a different kind of event, and the data gives them a different kind: each row is either a fresh app or a seeded repair. Codex’s overflow came down to width:auto on a hidden unit radio control. DeepSeek’s other app had an inline display:none that survived a valid calculation, so it worked out the result and kept it to itself. GLM’s other app gave the right number of paint tins and never showed the net paintable area after subtracting openings. Each was repaired in its own session, starting from the failing app, and served from a new deployment, and all three repaired apps passed.

None of that touches the five out of eight, top or bottom, because that fraction answers a specific question: what happens when an agent gets a fresh brief? A repair answers a different one, and its denominator is three. Add the two together and you get eight of eight, which is a score from an experiment nobody ran.

Even the repairs need two numbers. The record distinguished acceptance of the app from acceptance of the whole session, which also required that the session exited as completed. All three apps passed, and only two sessions completed and passed. The DeepSeek repair needed an operator to stop a browser process that belonged to the run before the client resumed and completed. GLM corrected its Rust compile errors and deployed the repair, but a provider request returned 401 before the final handoff. A working app and an unfinished session are separate results, and one column can’t report both. So the evaluation ended with four fractions rather than one:

The smaller decisions in the record format leaned the same way. Token counts and estimated costs were optional, because unknown is a different value from nothing, or as a comment in that code has it, “a 0.0 in a cost column is a claim”. Each record carried the git revision, whether the tree was dirty and a hash of the surface under test, so a later comparison could say what it compared. Missing attempts and infrastructure failures stayed visible instead of disappearing from the table. Eight apps from four client-and-model pairings are enough to find defects, which was the point of this round, and too few to rank the models or to call a release accepted.

Graders will keep being wrong, so the useful design question is what correcting one is allowed to touch. The record is what happened, the corrected observation is what you now think about what happened, and the repair is what you did next. Put those in one column and the last line of your report can say eight out of eight.

Chris Chabot, November 2026