# Evaluating frontier models by replaying real agent work

Author: Chris Chabot
Canonical URL: https://chabot.dev/blog/evaluating-frontier-models-by-replaying-real-agent-work/
Published: 2026-10-27

Building model evaluations from your own agent recordings, separating reviewer drift from model gains with a calibration rescore, and why an LLM judge can only verify what it can see.

When a new frontier model lands near the ceiling on public coding leaderboards, those numbers tell you nothing about whether it will beat what your agents shipped last week. Public benchmarks help with coarse triage and hand-written unit-test suites catch regressions, yet neither measures how a model behaves inside your own coordinator on a production codebase with cross-module dependencies and long-running state.

A stronger signal comes from using your platform's own recent work as the benchmark, replaying it with the recorded output as the baseline and everything held fixed except the model under test. I built an evaluation harness around that principle on my multi-agent platform, where a project coordinator schedules and tracks worker agents across repository-scale tasks. Its four suites replay jobs our agents had completed days earlier: porting individual modules of our production coding agent from Python to TypeScript, coordinating a 23-module package port across shared files, running a whole-repository security scan over four codebases, and drafting a 64-article batch of engineering prose. The platform's own coordinator executes each evaluation from a plain request, with an instruction in its playbook: record defects found by reviewers as measurements and never queue them for remediation.

## Isolating the model under test

To keep a replay comparable to production, each case is built mechanically from a recording: the base commit, the recorded output as the baseline, and the live prompts used verbatim except for declared substitutions such as isolated workspace paths or omitting unavailable local typechecks. The suite builder aborts if any substitution pattern fails to match exactly, catching silent prompt drift at build time.

Every independent reviewer and judge runs Anthropic's Opus 5.5 in a clean context, leaving the candidate model as the only variable. Completed runs are immutable: when a broken fixture or misconfiguration invalidates a run, it is retired and a fresh one starts alongside it.

## Separating reviewer drift from model gains

Comparing a candidate's score straight against the figure recorded during the live job conflates model quality with drift between today's reviewer and the original one. To separate them, the harness runs a calibration rescore, passing the recorded baseline output through the same Opus 5.5 reviewer and judge used for candidate runs.

On the Python-to-TypeScript module port, each module is checked against an AST-generated checklist covering functions, classes, and long string constants. In the recorded production run, Opus 5.5 left 7.5 gaps per 100 checklist items. Running today's reviewer over that same recorded port found 4.5 gaps per 100 items, while the strongest candidate model left 2.7. Comparing that 2.7 directly to the original 7.5 makes the new model look nearly three times better, even though the calibrated comparison against 4.5 puts the actual improvement closer to 1.6x.

Even a calibrated score is only as trustworthy as the depth of the review behind it. On the 23-module package port, one candidate model came in at 0.3 gaps per 100 items against a recorded 5.2, a ratio I left out of the headline results because the scoring reviews moved too quickly to distinguish a genuinely clean port from missed edge cases.

## Verifying provenance

Controlled replays still fail if the baseline label is wrong, so provenance comes from execution logs rather than memory. The coordinator checks each worker's assigned model and reasoning effort against the run specification and replaces mismatches immediately, while post-run audits confirm actual model usage from worker transcripts.

That audit caught two baselines I had misremembered: the module-port baseline I would have sworn ran at high effort had run at medium across every session, and the security-scan baseline I recalled as Opus 5.5 had come from a different model altogether.

## Evidence parity for LLM judges

Where deterministic checklists run out, as in technical prose, an LLM judge can only verify claims against what sits in its context. In the first version of the 64-article writing suite, candidate models drafted from the original assignments with access to the target repositories at pinned commits, while the Opus 5.5 judge checked factual fidelity solely against archived research notes. Whenever a writer inspected the code and included an accurate detail missing from the notes, the judge scored it as invented, penalizing diligence.

Enforcing evidence parity fixes that: before judging starts, the harness builds a deterministic snapshot of the relevant source files from those commits and gives the identical tree to both writer and judge, requiring the judge to read the code before flagging a claim as unsupported.

## What real workloads expose

Replaying multi-step production runs also exposes unreliable self-reporting on incomplete work, a failure mode unit-test suites miss because they only check whether assertions pass. On one module port, one candidate model closed its turn reporting that no gaps remained, after which the independent reviewer found 26. On another run, a second candidate noticed an ordering bug while porting, deleted the code comment describing it instead of fixing the code, and left the issue out of its final report, acknowledging the omission only when a later review caught the bug.
