Grading agents without trusting the simulator

To find out whether an agent can operate a platform through its tools, you need somewhere it can do damage without consequences, and that usually means a simulator. The simulator holds the state, applies the transitions and knows what happened after every call, which makes it very tempting to let it also say whether what happened was correct. Once it does, each of its bugs becomes part of the answer key. If a scale transition quietly drops the request and the expected state is computed by running that same transition, expected and actual agree, every agent passes, and the evaluation reports the bug as agents succeeding.

At the end of August 2026 I built an experimental lab around that problem, working with an agent. The subjects were two builds of the MCP server for Ferrite, my own platform, so the whole arrangement ran an obvious risk of confirming what I hoped to see: my server, my simulator, my grader.

Two owners

The structure starts with a plain ownership split. The simulator models the platform’s nouns, applies calls to them and injects typed faults: timeouts, retryable 503s, 409 conflicts, an ambiguous acknowledgement, a plausible false success and a stale cursor. Its only outward surface is a single observation function, it reports a divergence when a call falls outside what it supports, and by design it never judges whether its own behaviour was correct. The oracle takes four inputs, the scenario’s written expectations, the calls that were observed, the state before and after, and the tool catalog, and it never calls simulator transition code. It therefore cannot ask the simulator what ought to have happened. That is written down separately, as predicates over what can be observed, so in the scale example the oracle expects three running instances, reads back one and says so, where shared code would have agreed with itself.

The oracle still reads the state the simulator reports, so a transition that writes a plausible wrong value can satisfy a predicate written with the same wrong assumption. What the split removes is the cheapest failure, in which the grader agrees with the simulator because they are the same code. It also leaves the grader with no dependency on the simulator, since it needs only calls, observations and a catalog, so the same oracle can grade a live server, and a result that differs between a simulated run and a live one can become evidence about the simulator.

A grader that has never failed anything hasn’t been tested either, so the oracle gets an adversary of its own. A mutation library holds thirteen known-bad variants, which it must catch, and five acceptable ones, which it must let through, and a scenario is eligible to become a gold scenario only if all of those come out right. That checks sensitivity and specificity at once, in the spirit of metamorphic testing. The reason for a failure matters as well. The variant that reaches into another account must fail through a safety veto and never merely through the goal predicate, because an agent that damages someone else’s app and also misses its goal would otherwise look like an agent that just missed its goal. Six scripted reference agents supply known behaviour to calibrate against.

Ten gates

The verdict is ten gates rather than one bit, because “the tool ran”, “the task got done” and “the agent stayed within its permissions” are different claims. An infrastructure failure blocks everything, since it says nothing about the agent. Calls are checked against the tool contract and then against the scenario’s checkpoints, with their dependencies and bindings, and a consequence that can no longer be reached is marked blocked rather than failed, so a single corrupted argument produces one root cause at the type check instead of a page of failures downstream. Errors are graded by effect: a result in a success envelope whose structured content is nothing but an error object counts as an error, whatever the server says about itself. A set of minefields covers an agent creating a new app when it shouldn’t, mutating another account, reporting an app as running before any terminal status, acting destructively without consent, repeating a non-idempotent mutation, disclosing a credential, inventing an argument or calling a tool that doesn’t exist, and four security vetoes apply even when a scenario lists none. Argument provenance is conservative: a value fails only when it is absent from the legitimate sources, such as the task and the answers to a clarifying question, and when the grader doesn’t have the task it abstains.

What the agent says it did needs a gate of its own. The claims gate is lexical: whole-word matching, clause splitting and negation scope. “Validation ran; the app is NOT deployed” isn’t a deployment claim, and “Tools for running-status operations were unavailable” isn’t a claim that anything is running. “my client is running in read-only mode” is a remark about the environment, while “my client is running and the app is live” makes that same remark and also a claim about the app, and the remark can’t be allowed to hide the claim. After a failed call, “I couldn’t deploy it: the ferrite cli is unavailable” is an honest report, whereas “the deployment did not fail; everything succeeded” mentions failure only to deny it, and an acknowledgement has to actually affirm the failure.

The third answer

The most consequential decision is that the oracle is allowed not to know. A harness with only pass and fail has to round every missing fact to one of them. Here, a predicate such as status != failed evaluated against a path that isn’t there returns nothing at all, which the code describes as “unobserved, not PASS”. A gate that answers N/A without having been declared optional doesn’t pass. A scenario with a cleanup plan judges its goal at the observation taken before cleanup, because a correctly deleted app would otherwise look like a failure to deploy one, and if that observation is missing the goal is UNVERIFIED. An agent that stops without a single call to ask a clarifying question gets UNVERIFIED on its route and an entry in the unresolved list, rather than a failure. Unknown effects go to review, and they stay unknown until someone reviews them.

In one matrix with a live model as the client, two cells stayed that way after every attempt ran out of its step budget, at budgets of 10 and 24, having made only read-only calls and no final claim. The verdict stayed UNVERIFIED, with the agent’s failure to terminate under budget recorded separately. Scoring them as failures would have tidied up the table and would also have been wrong, because inventing a verdict from an interrupted trajectory is a fabricated claim about the agent.

The same discipline applied to a model-based judge I tried for the softer qualities of a run. Across 234 scored cells and six dimensions, not one dimension was admitted. Four moved their mean score by more than 0.5 when the presentation order was swapped, and one reached an agreement coefficient of 0.42 against a floor of 0.60, where the reference reviewers agreed with each other at 0.67. So the judge has no effect on any verdict and its abstentions are routed to expert review.

In a separate campaign against real deployments, the grader misread one client’s tool names and failed to recognise browser control. When those checks were fixed, the original records and their failed verdicts stayed as they were, with later checks joined on by hash and repaired apps given new deployments, so that a fix couldn’t count as a fresh success.

None of this makes the grader right. It makes the grader’s errors its own, visible and separately fixable, and it leaves the evaluation able to report that it doesn’t know.

Chris Chabot, November 2026