# Letting an agent diagnose production without giving it a shell

Author: Chris Chabot
Canonical URL: https://chabot.dev/blog/letting-an-agent-diagnose-production-without-giving-it-a-shell/
Published: 2026-10-16

Why read-only is not enough for an agent-facing operations interface: bounded verbs, stated uncertainty, fenced tenant text, analytics checked against ground truth, and leak-proof suppression.

When a support ticket comes in, the question you actually want to ask an agent is "a customer says their todo app is broken: find it and diagnose it." Investigations like that cut across accounts, billing, deployments, logs and edge traffic, which is the kind of multi-system triage agents handle well. The fastest way to wire one up is a shell on a production jump host, SQL or log-query passthrough, or your own operator credentials. That works right up until the agent mutates state, runs an unbounded scan, pulls personal data it never needed, or acts on a prompt injection buried in tenant-written text. The cautious alternative, a handful of canned report endpoints, avoids those risks at the cost of failing on the first follow-up question.

Simply making the interface read-only solves less than it looks. An agent investigating production needs a view it cannot damage and that cannot mislead it. Read-only stops writes, yet says nothing about data exposure, query load or prompt injection, and it misses the quieter failure underneath all of them: analytics that agree only with themselves. Getting that right comes down to least privilege and data minimization paired with independent ground truth and stated uncertainty. I built this operations layer for Ferrite Cloud, my hosted platform for apps and agents, where a collector turns edge logs, MCP audit records, ledger entries and deployment state into signed, redacted documents for an operator console, and an OAuth-gated MCP server lets an agent query them.

## Checking analytics against independent ground truth

Locking down an agent's read path buys nothing if the analytics behind it are wrong. The console's visitor funnel passed every test because every test compared the console against the collector's own output. Meanwhile, the funnel missed any visitor whose only traffic was MCP rather than browser page views, an awkward slice of the audience to overlook on an agent platform.

To break that self-referential loop, a reconciler reads raw nginx edge logs and the MCP audit trail in Google Cloud Logging outside the collector pipeline, then verifies the collector's stored events, published documents and console API against a separate model of the data contract. Alongside it, a labeled synthetic outsider walks the funnel end to end, covering browser pages, installs, OAuth discovery, MCP sessions, deliberate errors and app deployment, and asserts that its own visitor record matches every step. Verification networks and accounts carry their own classification so test traffic stays out of external metrics instead of inflating the conversion funnel it checks.

The first live run found six defects that internal tests had missed. A 240 KB document cap was silently truncating a week's tables; MCP tool calls were double-counted as both the tool record and its enclosing request; and the Ferrite CLI went unrecognized because it opens with `tools/list` rather than `initialize`. After the fixes, the synthetic run passed 80 of 80 steps, and both checks now run at release acceptance.

## Why read-only is not enough

Removing write permissions still leaves an agent able to run expensive queries, reach internal hosts or pull personal records it does not need unless the read surface itself is bounded. The MCP server exposes 31 strictly read-only tools built from seven composable verbs: `resolve`, `list`, `get`, `timeline`, `metrics`, `logs` and `aggregate`. Consistent identifiers, cursors and time windows let one call's output feed the next without glue code, so an agent can follow an investigation wherever it leads without needing ad-hoc query access. Arguments are enums, validated identifiers, bounded windows or literal substring filters over already-redacted rows; no SQL, query language or regular expression crosses the boundary. Structurally, the server is a sink that reaches no host or backing service, reading only the signed, redacted documents pushed by the collector.

Each tool call writes an audit record before touching data and fails closed on an audit error. Budgets per token, client and subject are charged before lookup or validation, preventing a spent token from holding in-flight slots through repeated denied calls. Personal columns require a separate 900-second OAuth scope with its own row budget; without that scope, those columns are dropped from the response and named as omitted.

## Stated uncertainty and untrusted text

Verified data can still mislead an agent if the API flattens missing values into zeros or mixes untrusted strings into its own output. Every response from the MCP server states how the answer is known: its sources, collection staleness, truncation and any withheld fields. Unavailable data is never encoded as zero or an empty list. An empty result from a degraded collector looks identical to quiet traffic once an agent summarizes it, so an explicit unavailable state keeps a missing number from being reported as none.

Tenant-written text creates another way to steer the investigator, since an app name or request path can carry a prompt injection. Every string a tenant or visitor could have supplied returns fenced inside a labeled wrapper, stripped of the control, direction-override and Unicode tag characters used to hide instructions, and is never composed into the server's own sentences.

## Preventing aggregate leaks by subtraction

Data minimization also breaks in a subtler way once an interface exposes aggregate tables. Withholding any bucket below a threshold is the standard defense against exposing individual accounts, yet in isolation it fails to basic arithmetic. Take an illustrative country breakdown with a suppression floor of five accounts: the table publishes a total of 50, lists 28 in the US and 19 in Germany, and suppresses the UK row. Subtracting the published cells from the total recovers the suppressed 3 immediately.

Complementary suppression closes that single-table leak by withholding a second cell, here Germany, leaving only the US count and the total visible so neither missing row can be derived from the total. That protection still breaks across boundaries if a linked table, or a second document published for the same time window, exposes a subtotal that lets you compute the hidden cell. The collector therefore applies complementary suppression across linked tables and across both documents it publishes for a window, and a property test subtracts every published pair to confirm that no suppressed cell is recoverable.
