# Letting agents observe and maintain their own platform

Author: Chris Chabot
Canonical URL: https://chabot.dev/blog/letting-agents-observe-and-maintain-their-own-platform/
Published: 2026-10-22

How to give agents access to the platform they run on: broad redacted reads, narrow self-referential writes, an explicit unknown for what is deployed, and reversible changes.

If you operate an autonomous agent platform long enough, you end up SSHing into a server to paste logs into an agent's chat thread so it can diagnose the system it is already running inside. Across two projects, we did that 25 times. A dead build lease would stall builds for hours, or a sandbox would be replaced mid-run, and the agents doing the work had no way to see what had happened.

Giving agents a shell on the host is the obvious shortcut, and a reliable way to turn a confused retry loop into an outage. Keeping humans in the middle as the observability layer avoids that risk, though it defeats the point of autonomous agents. The workable boundary is asymmetric least privilege: broad, redacted reads across the stack; narrow, typed, self-referential writes that act only on resources an agent can name from its own listings; and shared behavior changes restricted to compare-and-swap with exact undo.

I built this boundary into a hosted coding-agent platform whose agents develop and operate the system itself. Each agent session runs in an isolated sandbox behind a central gateway, and agents reach that infrastructure through two surfaces: a runtime capability built into each agent's own execution environment, and an admin MCP server for authorized platform-wide operations.

## Seeing its own runtime

The first surface lives inside the agent runtime. An agent can inspect its own alerts, sandbox state and replacement history, build lease, active jobs, execution lanes and logs, all redacted before reaching the model.

Recovery on this surface is strictly self-referential: every action accepts only identifiers returned by the agent's own status listing, never caller-supplied paths or process IDs. Reclaiming a stale build lease requires the current lease ID and evidence that its holder is dead; replacing a degraded sandbox requires an independent failed health probe. That keeps local recovery bounded to what the process actually owns, without putting an engineer in the approval loop.

Logs and events live in pre-sized ring buffers, so recording diagnostics under memory pressure never grows the heap. Every probe runs with a deadline and a typed status, turning a full disk or a hung call into an alert rather than an exception that breaks the reporting path itself.

## Seeing the whole platform

Local inspection covers one session, while production failures cut across sandboxes, the gateway and the host. Authorized agents inspect the rest of the stack through the admin MCP server, which exposes roughly 190 tools. A single forensics call pulls a thread's transcript, cost, compaction state, sandbox history, gateway logs and traces; other tools expose failed host service units, the kernel log, gateway event-loop lag and a release ledger of deployed commits.

Because those reads span the entire platform, identity and scope are checked at the edge and re-verified on every call, arguments are hashed into an append-only audit log, and responses are redacted. Agents also act under their own verified identity rather than sharing operator credentials, keeping autonomous actions distinct from human ones in the audit trail and letting the server block destructive operations like project deletion for coordinator agents.

Broad read access comes with a corollary: evidence an agent cannot reach does not exist for an autonomous system. Our heap-growth alert originally asked an engineer to capture a memory summary on the host, which worked fine until a breach fired after midnight with nobody at a keyboard. Writing the capture to disk automatically still left the summaries on a host filesystem agents could not inspect, so the breach handler now writes two memory summaries ten minutes apart and logs what grew directly into agent-readable telemetry.

## Knowing what is actually running

Before an agent acts on a failure or verifies a fix, it has to know what code is executing rather than what landed on main. The platform answers a direct question: is commit X running in the gateway, or in thread Y's runtime? Instead of inferring from deploy timestamps, the check tests commit X by git ancestry against the commit each gateway or thread runtime was released from.

The query returns yes, no, or unknown. If a runtime predates the release ledger, or git cannot place the commit in the ancestry graph, the answer is unknown, never guessed. In an autonomous loop, unknown has to be a first-class result. If a boolean check defaults missing provenance to no, an agent restarts healthy runtimes or re-applies fixes that are already live; if it defaults to yes, a stale deployment passes inspection. An explicit unknown prevents missing history from turning into an unverified mutation.

## Changing its own code

Once agents can inspect the platform and verify what is running, they can also develop and ship changes to their own code. To experiment without disrupting live work, an agent builds a pushed commit into an immutable trial candidate while existing sessions stay pinned to their current version. Candidate runs are evaluated by comparing P99 latency through bootstrap resampling, with timeouts treated as censored observations. Promoting a winner uses compare-and-swap with readback, and rollback executes an exact compensation that never overwrites a later selection. The lifecycle runs through a write-ahead effect journal whose state transitions are checked against reducers exported from a model in Lean, the theorem prover.

The same operating model extends past runtime trials to repository-wide maintenance. We pointed the platform's own scan workers, built on DeepSec, an open-source security-review engine, at our main repository. The scan inspected nearly 8,000 files and confirmed each finding against source code. Over two days, agent workers landed hundreds of fix commits through the normal release path, each citing the findings it closed.

To verify that work without inheriting the first engine's blind spots, we ran a second, independent security-audit tool with the initial scan's findings withheld. When that pass hit its budget before covering the entire repository, it recorded its own coverage as incomplete rather than reporting a clean result.
