Did your agent experiment test what you think it tested?

Until 2 August 2026, my notes on reasoning retention contained a tidy finding: stripping a model’s earlier reasoning from the conversation made GPT thrash 2.7 times as much. Something had certainly been measured. The probe behind the fast arm, however, never wrote the main-model pin, so every fast-probe row labelled GPT before that date had run on the default main model. The finding described a model that had never been in the main loop, and I retracted it.

Nothing about a row like that announces the problem. It carries the right label, and in a results table the label is all you see. Two weeks later a different report recorded the same species of error: every measurement before the third wave of a hill-climbing campaign had run on a different agent from the one under test, because the campaign’s runner silently defaulted to it against localhost. Those numbers were perfectly good descriptions of something else.

So before asking whether a change helped an agent, I want to know whether the comparison actually contains the change. Most of the evaluation tooling I built over July and August is an answer to that question, and the ways of getting it wrong sort into a few distinct kinds.

The wrong model

The runner I built for paired experiments treated an arm as a claim to be checked. An arm it couldn’t parse stopped the run rather than falling through to a default, every condition had to be fully labelled, and each setting was written at project scope and then read back. The read-back had to understand one awkward case: asking for a value equal to the default stores nothing, which is correct behaviour and looks exactly like a failed write. Model and deployment identity stayed attached to the results, and a run that fell back to another model was ruled ineligible for the comparison. Eligibility never touched the reward, so a contaminated run kept its score and simply didn’t get a vote. The GPT probe was changed to apply the full evaluation profile, pin included, and the GPT conclusions were narrowed to a pinned, paired comparison on tasks from Datacurve’s Deep-SWE benchmark.

The feature that never ran

That hill-climbing campaign also had an arm meant to switch on edit batching, and it did nothing, because the setting was resolved when the module loaded and was out of reach by the time an arm tried to change it. The report puts the consequence better than I can: “A no-op arm returns a confident null result”. A null is the dangerous outcome here, since a surprising win gets scrutinised and a null invites everybody to move on.

This is why the paired runner also harvested telemetry for the mechanism itself on every turn, keeping the last snapshot per turn and recording counts and bytes only. When it couldn’t establish that a mechanism had taken effect, it recorded the effect as unknown rather than assuming success. The same telemetry stops a result being over-read. One before-and-after report from mid-August is headed as a measurement of the full multi-agent session, and its runs recorded zero compactions and zero subagents, with working sets disabled. Whatever it shows about cost, it can show nothing about compaction, routing or orchestration, because the runs never exercised them.

The workload, and everything else that moved

The dated reports kept task identity, budgets, tool and model calls, polling, runs voided by infrastructure and feature configuration side by side, because a comparison across a changed task mix can’t be read causally, any more than the wrong-agent runs above can. Holding the tasks still doesn’t settle it, though, as that mid-August report shows. It compared the same three hard tasks, boa, cattrs and opa, on the same repositories and harness, chosen deliberately as two baseline passes and one near-miss. Passes went from two to three, tool calls from 982 to 421, terminal polls from 543 to 47, and the always-present prompt prefix from about 51,000 tokens to about 17,000, by moving rarely needed tool definitions and guidance out of it. On the two tasks with a cost in both runs, the recorded cost fell from $134.62 to $51.45. Boa has no baseline cost, so it stays out of that number.

Those are good numbers and they still can’t be carved up. Prompts, lazy tool loading and terminal waits all changed together, so the polling drop, which went straight at the thing the new terminal wait replaced, is the only part with a simple attribution. As for the headline, the only pair that changed verdict was boa, which had been failing by one hidden test. An exact sign test looks only at discordant pairs, and one discordant pair can’t be told apart from a coin toss.

The platform underneath can move as well. The first two waves of the hill-climb ran before a platform lag defect was fixed, so their failure rates of 20 to 53 percent were dominated by the platform rather than by model behaviour. They remained valid arm against arm, since both arms ran under the same defect, and they were useless as absolute numbers. The one arm-level win in that report, resident file views, moved edits per file from 1.82 to 1.53 and polls from 17 to 6 on the same tasks and the same build, which is what a comparison looks like when little else has moved.

Noise needs a number too, and that one came by accident. A duplicate launch ran the same cattrs configuration twice, at $8.34 and $9.19, which showed about 10 percent variation between two nominally identical runs. Against that, a broad new operating mode came in at $11.97 with 76 calls and 69 percent more input. One run against a baseline and its accidental twin is thin evidence, but it was thin evidence pointing the wrong way on an expensive feature, so the flag went off and the mode was never written up as a win.

The feature that ran and didn’t help

The hardest results come from experiments where everything works. I built a replay laboratory for context compaction that froze real histories, tested fold boundaries and retrieval of the omitted middle, and compared model roles under blinded judges. What a replay can’t tell you is whether a deployed agent gets cheaper, so at the end of August a paired follow-up ran three tasks, DynamoDB, Koota and Pest, one attempt per arm, on two deployed builds with verified package hashes. The mechanism improved, with semantic folds going from two of seven to six of eight. The outcome didn’t. Both arms passed two of three, with Pest failing on the same test in each. Summed agent time rose from about 253 minutes to about 277, tool calls from 346 to 371 and cost from $61.51 to $80.49, with Koota alone going from $9.93 to $18.66. After folds, adjacent provider requests used 37 to 58 percent fewer input tokens. Those would have made a fine slide. The report records no efficiency gain across the completed tasks instead.

The arithmetic comes last

Only after all this does it make sense to talk about statistics, and the tooling there was deliberately small. For paired experiments, the analysis implemented what the pre-registration asked for and nothing more: a paired two-by-two table, an exact two-sided sign test on the discordant pairs, raw per-pair values and median paired ratios for continuous outcomes, and seeded 10,000-resample bootstraps for 90 percent intervals, so that anyone rerunning it got the same interval. Missing pairs and zero denominators were named rather than dropped.

The instruments deserve the same suspicion as the agents. The failure inventory I kept for one mid-August campaign has entries for truncated receipts, pipelines whose exit codes were masked, verification that verified nothing and working context billed repeatedly, and it counts four occasions on which the instruments themselves lied. Each entry maps to a fix and to something that would catch a regression: a unit test, an operational rule or a behavioural assertion run against real task runs.

If a row says GPT, you should be able to show that GPT ran. If it says compaction, you should be able to count the compactions. And if both arms pass two of three and one of them costs nearly a third more, the report should say so in those words.

Chris Chabot, November 2026