Proving the harness before scoring the agent

The experimental controls a coding-agent evaluation needs at each layer, from task certification and configuration read-back to audit tooling and grader corrections that never overwrite the run.

August 20264 min readEvaluationVerificationAgent systemsMeasuring agents before they ship

Thin ochre and iron red stains meet rough knifed blue-black and grey-green swipes along a jagged front of crossing strokes and half-buried chalk touches.
On this page
  1. Positive and negative controls on every task
  2. Manipulation checks for configuration and mechanism
  3. Known-positive checks on audit tooling
  4. Immutable run records and derived grader corrections

When an agent benchmark row comes back red, the reflex is to charge it to the model, even on a task whose reference solution would have failed on the same machine. A surprising null result in an A/B run is dropped just as quickly, without checking whether the change under test ever executed. In early agent evaluations, the harness is the likelier culprit, whether through a missing toolchain, a silently defaulted model, a log endpoint that truncates, or a grader that misreads a tool name. Rerunning suspicious cases or skimming transcripts catches a bug once and leaves the method unchanged.

Standard experimental control applies at every layer: positive and negative controls on each task, manipulation checks confirming the treatment ran, known-positive checks on audit tooling, and immutable run records with grader fixes stored as derived views, as in event sourcing. I implemented these controls across two evaluation harnesses. The first is a TypeScript runner for a production coding agent, which works on repositories inside cloud sandboxes; it creates a project through the real agent platform, assigns a task from Deep-SWE, Datacurve’s public software-engineering benchmark with hidden tests, and verifies the files the agent actually produced. The second drives Codex, Claude Code, and Pi as real coding clients over MCP to build and deploy small web apps on Ferrite, the cloud platform I built, then checks those deployments in a browser.

Positive and negative controls on every task

Before an agent runs, two agent-free controls certify each task on the host: the reference solution must pass the hidden tests, and an empty patch must fail them. On one macOS host, the positive control certifies 84 of 113 Deep-SWE tasks; the other 29 fail before any agent is invoked, owing to environment mismatches such as missing runtime dependencies or container-assumed paths. None of those 29 can produce a verdict on an agent, and running models against them only buys false negatives at inference cost.

Those two controls certify only two inputs: an untouched base repository and the reference patch. A real coding agent does more, notably writing its own tests to verify a fix, which is good practice until the verifier applies a hidden test patch over the same package. On a task in dasel, an open-source Go tool, an agent’s new test files collided with the hidden test patch and the run came back failed. Excluding agent-created test files before applying the hidden suite requires two predicates: the file must be untracked at the base commit and match test-file naming patterns. Tracked files stay untouched, so edits that weaken pre-existing tests still count. With that filter, the dasel run moved from a reported failure to passing all 146 hidden subtests.

Manipulation checks for configuration and mechanism

A certified task proves nothing if the experiment arm never delivered the treatment under test. In multi-layer agent platforms, an unparsed arm flag or a silently defaulted model rarely raises an error; it just produces a clean table for the wrong hypothesis. Without delivery checks, I recorded and later retracted a finding that stripping reasoning made GPT thrash 2.7 times as much. A probe script had never written the model setting, so every run labeled GPT had executed against the default model instead.

To prevent that, every arm fails closed on an unparseable specification rather than defaulting, writes its configuration to the platform and reads it back before execution starts, and records which model actually served the requests. Alongside the task verdict, per-turn telemetry counts whether the mechanism under test executed. When a feature never fired or a fallback model served a turn, the run is flagged ineligible and excluded from the comparison while leaving the underlying pass or fail record intact.

Known-positive checks on audit tooling

The scripts that aggregate run logs and telemetry can fail just as quietly as the harness itself, because a low or empty count looks identical to a clean run. An audit tool therefore needs a known-positive check before an empty or shrinking count is trusted. On one set of runs, a log endpoint capped responses at about 100 records without signaling truncation, so an audit reading it reported 4, 5, and 7 events where the true counts were 11, 10, and 13.

Immutable run records and derived grader corrections

Even with certified tasks, verified configuration, and checked telemetry, the grader itself can be wrong. Overwriting historical run records after a grader fix destroys the audit trail, making it impossible to separate a better agent from a repaired grader or a patched artifact. Original run records and transcripts stay immutable. Any parser fix or updated browser check sits beside the original record as a derived view, bound by SHA-256 hashes of the source record and transcript and citing the specific transcript events that justify the revised verdict.

In an eight-app cohort on Ferrite across Codex, Claude Code, and Pi, initial automatic grading passed four of eight deployments. Two Pi runs had failed because the transcript parser missed Pi’s direct MCP tool-name format and recorded zero MCP calls even though the retained transcripts documented successful tool execution and deployment; fixing that parser, along with a brittle form-validation check on one of the two apps, flipped both unchanged deployments to pass. In the other direction, a stricter mobile viewport check caught layout overflow on a Codex app that had initially passed, flipping it to fail. Those three grader changes moved the fresh-build count from four passes to five of eight, leaving the original verdicts and zero-call counts intact in the base records.

The three apps with genuine defects went through separate repair sessions that produced new deployments. All three repaired deployments passed browser verification, though only two of the three repair sessions completed, one of them with operator intervention. Those outcomes stay in a separate repair count and never enter the fresh-build denominator, so five initial passes plus three repairs cannot turn into eight of eight.

Chris Chabot, August 2026