When the evaluator is the broken part

On 2 November 2026 we put sixteen combinations of coding agent, model and setup through a dress rehearsal on a fresh installation of Ferrite, the application platform I build on and operate. Each one built an app, and all sixteen apps passed independent CRUD and digest checks. The full verdict for those same sixteen runs was two passes and fourteen failures. Both results are correct. Reading them together depends on seeing that a benchmark like this asks several different questions at once, and that one of the things that went wrong that day belonged to our test runner rather than to anything an agent had built.

The first question is what the agent produced: whether there is a deployed application, whether it creates, reads, updates and deletes what it should, and whether something other than the agent can confirm that. The second is how the agent got there. These cells exist to test what it is like for a coding agent to use the product, so they come with rules about which tools it should use, and an app built by some other route would say little about the experience we set out to measure. The third is what the observer can actually prove about the first two, which is a property of the evaluation and not of the agent. An evaluator that folds these into one column will report a broken product when its evidence is merely incomplete, or a healthy product when the agent went around it.

On the first question the answer was sixteen out of sixteen. The clock gave a slightly different count, with fifteen cells inside the original 420-second limit and one at 482.1 seconds. We later authorized a 10 percent performance tolerance, which moves the line to 462 seconds, and a limit that moves after it has been set deserves some suspicion, particularly from the person moving it. In this run it changed nothing, because fifteen cells did not need it and the sixteenth is outside it as well. The runner now reports strict and tolerated timings separately regardless, so nobody has to work out which limit a green number was measured against.

The fourteen failures were all on a separate verdict, tool observation, which asks whether the record of a run shows the agent keeping to the tools its cell allows. None of them says an app was broken. Someone who sees two out of sixteen and concludes that the agents mostly failed to build anything has misread the result as badly as someone who sees sixteen working apps and concludes that the benchmark passed.

The URL in the prose

One of the day’s problems really was the evaluator’s. A fork check seeds an app with known data and then compares the state of the original and its fork, so the runner has to know which app to seed. Our runner, which I developed with agents like the rest of this harness, was getting that from prose, by selecting a URL from the text of the run. In four cells, all on the same model, the URL it selected belonged to a diagnostic fork, so the check seeded the wrong app. Whatever it reported after that was a statement about the wrong app, recorded against the cell as though it described the right one, and anyone reading a bad result there would have taken it for an application failure.

A URL in prose is something a writer happened to mention, and an agent explaining its work may mention several. Once a grader reads narrative to find its target, the way a model writes can affect what gets tested, and a defect in the grader can then look like a finding about that model. The repair was to stop reading prose. The runner now binds to deployment receipts, the record the deployment itself returns, so the app under test is named by the system that created it. It also distinguishes a remote test failure from a deploy failure, which send you to different parts of the system. The guide we run releases from allows a broken check the smallest necessary repair with the original failure retained, and that is what this was: the original receipts were not touched.

The same pass found a different kind of problem in what we were handing the agents. The published page starter contained a stale store example, oversized compared with the canonical plugin. We brought it back into line with the plugin and corrected the guide we were serving. So the list of suspects for a red cell has at least four names on it: the agent, the application, the grader and whatever we gave the agent to read.

What the reruns could not fix

We then ran exactly two selected successor runs. They finished in 278.3 and 410.6 seconds, both under the original 420. Both passed the CRUD checks, the original and fork state checks and the source composition checks, and both retained a failed tool-observation verdict.

That last part shows exactly how far the repair reached. The runner was fixed and the fork checks were green, and none of that touched the remaining red. No prohibited tool use was observed in either run, and it would be very easy to write that down as “the agent used only the permitted tools”. Those are different claims. An observer working from the record of a run can catch a prohibited call if it appears there, and it cannot show that nothing happened outside what the record contains. We know how a record can miss things: in August a grader’s parser failed to recognise one client’s tool names and recorded zero platform calls for sessions that had plainly deployed. Proving exclusive use would take something stronger than reading the transcript, such as removing the other routes from the environment or observing them from outside the client, and that is a different experiment, which cannot be added to a run after it has finished.

A broken grader and a limited observer need different treatment. The wrong URL was a defect with a cause and a fix, and after the fix the check gives a different answer. “Exclusive use not proven” is an accurate statement of what we know, and the way to change it is to collect better evidence, not to reinterpret the evidence we have.

There is an obvious move available at this point, which is to go back and relabel: the apps worked, the fork defect was ours and no prohibited tool was seen in the reruns, so call it sixteen out of sixteen. No run ever produced that number. The fourteen failures stand, and the successor runs are a selected pair graded by a repaired runner, so they can’t be averaged into a sample they were never part of. The installation has gone too, because I asked for it to be taken down later that day to stop the test costs, which means nobody can go back and look again. What remains is the receipts, and they are only worth anything because nobody has edited them.

An honest report of that day therefore takes a sentence rather than a number: sixteen apps worked, fifteen were on time, two runs met the full contract, four fork checks were our runner’s own mistake, and two reruns on the repaired runner passed everything except the claim we still could not prove.

Chris Chabot, November 2026