# Whose mistake did the coding-agent benchmark catch?

Author: Chris Chabot
Canonical URL: https://chabot.dev/blog/whose-mistake-did-the-coding-agent-benchmark-catch/
Published: 2026-11-03

Reference and empty-patch controls, test-file collisions and environment faults: how to adapt a coding-agent benchmark and tell an agent's failure from the harness's.

A coding agent saying it has finished tells you very little, and a transcript in which it appears to finish tells you only a little more. The evidence worth having is the files it left behind, run against tests it never saw. [Deep-SWE](https://github.com/datacurve-ai/deep-swe), from Datacurve, supplied the ingredients for my 2026 evaluation: 113 tasks across TypeScript, Python, Go, Rust and JavaScript, each with a repository pinned to a base commit, a hidden test patch, a reference solution and a script that decides the reward. I inherited all of that and kept the solutions and tests byte for byte. My part was adapting it to our agent platform so that a run went through the product’s own project creation and execution paths. The harness created a project the way a user would, through the browser with Playwright or over RPC, pointed it at the task’s base commit, let the agent work under auto-approval, and when the turn ended handed the files that came out to a verifier.

That last step is where a benchmark can start misleading you without anyone noticing. Upstream’s scripts assume a particular world, with the code at `/app` and the right version of everything already installed, and my verifier had to rebuild that world in a temporary staging directory. It created the expected layout, rewrote paths in the scripts without disturbing anything that merely began with the same characters, gave Python a virtual environment, handled GOTOOLCHAIN and GOSUMDB for Go, ran dependency setup and the tests under one time budget big enough for both, and treated the reward file written by upstream’s script as the verdict. Every one of those steps was code of mine standing between the agent’s patch and its score. So when a task failed, either the agent had got it wrong or I had, and a benchmark that can’t tell you which has only told you that something, somewhere, went red.

The first defence was a pair of controls that involved no agent at all. One applied the reference solution, which ought to pass, and the other applied nothing, which ought to fail. If the reference fails, an agent’s failure cannot yet be trusted as a verdict on its patch; if doing nothing passes, the check has not distinguished the requested change from the base. Either result calls the harness into question. On a macOS host the reference-solution control certified 84 of the 113 tasks. The other 29 ran into the host itself: the wrong Node version, scripts that insist on `cd /app`, tasks that expect `mongod`. That made 84 a statement about one machine rather than a judgement on the benchmark, and it marked out 29 results from that machine that nobody should read as a verdict on an agent.

Controls have a limit, though, which is that they only certify the harness for the two patches they apply, and neither of those looks much like what an agent writes. A conscientious agent adds tests for its own change. Those tests land in the same directories as the hidden tests the verifier is about to add, sometimes declaring the same names, and Go is especially unforgiving about this, since two declarations of one test name stop a package compiling and a package that doesn’t compile has no passing tests in it. The cleanest example I have is a run on dasel that was recorded as a failure, with the base suite exiting 1. The hidden patch replaced files whose helpers the agent’s other tests depended on. With its three newly created test files excluded from staging, the same patch passed all 146 hidden subtests and the base suite as well. The agent had done the work, and its own diligence had knocked over the test of that work.

The repair looks simple until you list what it has to leave alone. The verifier removed files that were both untracked at the base commit, the set [`git ls-files --others`](https://git-scm.com/docs/git-ls-files) reports, and named like tests. The naming condition kept the new source files that make up the agent’s actual patch. The untracked condition protected two things a cruder rule would have destroyed. An agent’s edit to a test that already existed stayed visible to the base check, because weakening an inherited test has to keep counting as sabotage. And a file like `base_test.py` in Mobly, which is named like a test but is the framework’s product code, was tracked and therefore left alone. Where Go symbols collided, the verifier renamed the candidate’s symbol using a tokenizer rather than a text search, which kept it from rewriting a string or a comment that happened to contain the same word.

The detail I find most instructive concerns the tool built to catch these problems. A rescoring tool took a recorded run and graded it both ways, with the agent’s tests kept and with them excluded, so that a difference between the two arms would point at the verifier rather than the agent. With the exclusion always on, its two arms became identical and the tool built to detect verifier artifacts could no longer detect one, so the exclusion had to become optional for that one caller. A repair to a grader can remove the very contrast your diagnostics depend on, and nothing about the repair will tell you so.

The verifier’s own environment can decide a verdict too, and a bare FAIL hides that just as well. On a Rust task from the pest parser, one measurement had the base check exiting 127 in both arms, which is what a shell returns when it can’t find a command. A separate measurement with RUSTUP_HOME and CARGO_HOME redirected came back as `base=1` and `new=1`. Pest expected a bootstrap binary under the repository’s own target directory, while CARGO_TARGET_DIR sent the build elsewhere. Removing that variable gave `base=0` and `new=101`: the existing suite now passed, while the new check still failed. A leaderboard column would record all three as failures. Only the last had cleared the base check; the remaining failure still needed its own diagnosis.

Attribution also depends on being able to go back and look at what happened. A run whose captured evidence went over the 64 MiB cap was marked ineligible, with its reward forced to zero, rather than scored on whatever had been kept. Evidence files got SHA-256 receipts and a count of what the sanitizer had redacted, so a later reader could tell what was kept and how much was taken out. And test processes ran in their own session, so that a timeout killed the whole process group instead of leaving something running to disturb the next check.

It is just as easy to confuse having a harness with having a result. I also wrote an adapter for Harbor that ran our agent on [Terminal-Bench](https://www.tbench.ai/), packaging the agent into each task container, giving it a small standalone context so that its local tools operated on the task’s workspace, and routing its model calls through our inference service. The full-suite command was configured for 89 tasks at five trials each, 445 trials on paper. That number describes a command. A score needs the actual run outcomes, with incomplete and excluded trials accounted for explicitly. Patch-level tests have an outer limit too: they can tell you a function behaves, but not that a deployed application works in a browser, which is why a separate harness of mine watched real coding clients build and deploy apps and graded source artifacts, admission, deployment, probes, browser behaviour, task completion and effort as separate layers.

A failure you can’t attribute is expensive in a particular way. You go and change the agent, its prompt or its tools to fix something that was wrong with your own test bench, and the next run can’t tell you whether you helped. Being able to read a red result and say whose it is comes before any of the interesting questions about how good the agent is.