Measuring and gating a platform's agent experience

How to benchmark the instructions, tool descriptions and errors a coding agent reads on an MCP platform, holding everything else fixed, and turn the result into a release gate.

October 20264 min readInterfacesEvaluationRelease engineeringRelease qualification for a cloud platform

Scarlet strokes run into a thick band of scraped deep olive and ash grey, surviving intact only at far left, with chalk showing through.

Our leadership asked a coding agent connected only to Ferrite’s hosted MCP server for a team-offsite voting page, and it took 27 minutes and 85 tool calls to get a live URL. Ferrite is a constrained cloud platform where an application is a single Rust crate linking only the platform SDK, built and deployed remotely; users and their agents connect over that MCP server, a CLI, or both. When an agent takes nearly half an hour on a straightforward request, the reflex is to blame build times or write more documentation. Neither touches the cause when every component passes its tests and the agent builds a working app eventually. Left to explore an unfamiliar system, a coding agent behaves reasonably on its own terms: it reads documentation it does not need, reaches for the platform’s application framework meant for larger services, deploys test builds, and sleeps in shell loops while builds run. All of that makes sense on a complex service, and burns twenty minutes on a voting page.

When users arrive through coding agents rather than reading quickstarts themselves, the text the model reads is the developer experience. Developer relations tuned documentation and onboarding for people; agent experience is the same craft for a new kind of developer, with a benchmark instead of a hunch. On Ferrite, that meant holding the platform fixed, varying only what the agent reads, and turning the result into a release gate like any latency budget.

Benchmarking what the agent reads

To isolate agent-facing text from infrastructure, I built a bench that replays the exact voting-page request against production inside a sandbox, behind a local proxy. The proxy logs every MCP call and overlays four surfaces: server instructions, a starter brief, served documentation, and tool descriptions. Production never changes during an experiment; a winning variant still has to be built into the server.

Placement matters as much as wording. My measurements showed that Claude Code truncates tool descriptions at about 2,048 characters, dropping anything past the cut. Guidance for small apps in a separate guide or near the end of a long description goes unread; putting a single-build path before that cut in the description of the tool the agent is about to call eliminated the framework detour and most of the documentation reads.

The proxy also lets you measure a server feature before building it. Call traces showed agents waiting out deployments with sleep 60 and noticing finished builds 30 to 100 seconds late. Rather than shipping server-side blocking on speculation, I had the proxy emulate a wait of up to 50 seconds on build status and measured the instructions relying on it; the server implementation followed only once the numbers confirmed the gain. With that text and the 50-second wait shipped, the same voting-page request goes live on one build in as little as three and a half minutes, with 15 MCP calls.

Errors as interface

Error messages sit on the same interface as instructions and tool descriptions. A generic validation error forces an agent to guess which argument failed and what the server expects, spending extra turns on a mistake the server already diagnosed. On Ferrite’s MCP server, argument refusals are generated from each tool’s published schema and name the tool, the argument, and its allowed type or range. Deriving that text from the schema served to clients prevents drift as tools evolve and lets an agent fix a malformed call in one step.

Gating releases with a client matrix

Text degrades quietly as tools, docs, and models change, so a one-off bench only captures a good week. To keep the gain from eroding, I turned the bench into a release matrix crossing two models with eight client setups: MCP only, CLI preinstalled, CLI bootstrapped from setup docs, and both together, each with and without a local Rust toolchain, for 16 cells in total. Every cell builds the voting app and then forks it for a follow-on change. A release fails if an MCP-only build is not live within 12 minutes or if any phase has more than one failed build.

Because the agent is inside the system under test, a live app alone does not prove the cell took the intended route. Given any gap in isolation, coding agents routinely fall back to host binaries, local toolchains, or operator credentials. Before any model call starts, each cell checks how executables resolve inside its sandbox and aborts on a mismatch, proving a no-Rust cell cannot reach a compiler and an MCP-only cell starts without the CLI.

Initial conditions still do not capture what happens mid-session, so a post-run audit inspects the transcript for every command executed, and those rules have to be tested in both directions. An MCP-only agent that falls back to the CLI must fail, while one that stays on MCP and checks its deployed page with curl piped to grep must pass. An overbroad rule on shell command substitutions once failed ten of sixteen cells solely on that curl verification; parsing read-only text pipelines explicitly keeps enforcement strict on prohibited tools without rejecting ordinary page checks.

Chris Chabot, October 2026