Following an agent failure down to the process

A long agent run on the platform I work on ended with a status of completed and a last message that read, “I stopped this turn after 3 command infrastructure failures.” The status said the turn had finished. The message said the work hadn’t. Anyone scanning statuses would have moved on, and anyone reading the conversation was left with the only question that matters: what, exactly, failed? Answering it takes a chain of joins, from the conversation to a deployment, a runtime instance, a tool call, an adapter, a terminal session and finally a process, and each join is a place where the evidence can quietly go missing. I followed that path through separate investigations between June and August 2026.

The thread

The first join is the one most easily skipped. A thread is bound to a particular deployment and a particular runtime instance, and when something looks stuck the natural move is to look up the service by name, find it healthy and conclude that the trouble is somewhere else. So the diagnostics I built start from the thread. The debug tooling and the gateway expose the runtime identity it is bound to, the lifecycle evidence for that instance and the state persisted locally, which lets an investigation separate a stale deployment pin from a missing instance from a turn that is simply still running. Those three call for different responses, and none of them involves restarting an unrelated healthy process. The diagnostics needed maintenance of their own, too: once persisted state moved to SQLite they had to follow, because a script still working from the old JSON assumptions would have been answering about state that no longer lived there.

The transcript

For tool calls, the evidence is the transcript, all of it. That run left a transcript of 1,140 tool calls and about 900,000 characters, including 181 command executions, 50 terminal checks and 25 infrastructure failures. The transcript showed several separate causes stacked on one another. The flag for a direct command path had been wired into the layer that builds the sandbox pool, while the adapter a command actually used was resolved somewhere else, and that path also lacked a managed lifecycle. One line in the agent’s own instructions steered caches to /tmp, defeating the native scratch directory. In 75 of the 181 commands the agent re-exported the same environment boilerplate, and in 143 it re-prepended a cd. Its attempt to end the turn was rejected three times because its plan was considered stale. The fixes addressed the wiring and lifecycle, corrected the instructions and added an environment parameter on the command tool and a read-only exemption in the loop policy.

The list of things that were not issues is as useful as the list of things that were. Five sub-agents had been spawned, which looks suspicious and wasn’t a problem. An alarming 206 plan updates turned out to be a miscount of 16. There was no out-of-memory condition and no CPU starvation. A trace that only accumulates suspects ends with someone fixing the wrong thing, so an explicit acquittal is part of the result.

Between the tool call and the sandbox sits an adapter, and even that leaves fingerprints. The direct adapter issued command handles with sixteen hex characters after the prefix where the other path used a UUID, so the shape of a handle in a transcript told you which path a command took. The other adapter also had no way to wait for a job or drain its events, which is why one command could take up to 47 polls on that path and a single call on the direct one.

The pane

Below the adapter, commands were serialized under a per-session lock into one shared tmux pane. The terminal agent typed the command with send-keys, pressed Enter and blocked on wait-for under a timeout, and completion could only be signalled by shell hooks at the next prompt or on exit. When the timeout fired, it sent a single C-c, removed the signal file and reported a timeout. If the foreground process doesn’t give the pane back, the next command goes into the same pane and times out as well, and so does the one after it. The minimal reproduction is trap '' INT; sleep 600 followed by echo hi. A separate June coding task ran for about 27 minutes, through 43 commands and 22 file edits, with six consecutive iterations in which every tool call failed, “Timed out waiting for command in session … after 20s”, and finally a turn that ended on repeated tool failure. A user reads that as an agent flailing. The cause was one pane, several layers down.

The recovery I proposed started by checking whether the pane had returned to its prompt before accepting another command. The pieces already existed: a cancel operation could read the pane’s process ID and kill the process group, a status check could report whether the pane was at its prompt, and tmux itself offered respawn-pane. The timeout path needed to use those primitives to recover or replace the pane before the next command was typed into it.

A separate load probe exposed another problem with dispatch through the shared pane. Dispatch had a budget of about 12 seconds. On an eight-core sandbox with every core pinned it ran out of that budget, while E2B’s own command API dispatched in about 100 milliseconds, and a bigger sandbox made the problem less frequent without removing it. The optional workaround was an adapter that talked to the same sandbox by ID through that API, changed only command dispatch and left the filesystem mount alone.

The quiet ones

The hardest joins are the ones where a layer reports something plausible instead of an error. Command output was cut off at a suspiciously exact 150 bytes, which makes sense once you notice it was passing through a 200-column PTY as base64, and 200 base64 characters decode to 150 bytes; standard error on that path was always empty. File edits failed with advice to use CREATE when the real condition was replication lag: across 33 traces and 624 edits, all 11 failures came right after a platform clone, and an identical retry succeeded. Neither symptom points anywhere near its cause. The compensations, nonce-bound output separators and a three-valued coherence verdict with a two-second deadline, were there to give the next reader an answer instead of a guess.

Repository clones behaved the same way. Five of six attempts on one public repository, across two runs in June, died with exit code 97, one of a table of stage codes in our materialization step, and the tar standard error we had captured and replayed was empty all five times. The working hypothesis was that tar had been killed by a signal, but confirming it required access to the coordinator host, so it remained a hypothesis. What I could change was how the failure surfaced. The mitigation made the clone retry three times with a five-second backoff, log the return code and throw when it failed, so the interface showed a red chip where it had shown a green success.

The instruments

All of this depends on the instruments, and they fail in the same way. The log endpoint silently clamped at about 100 records, so an audit counted 4, 5 and 7 overseer injections where the truth was 11, 10 and 13, and later read six edits as none. The instruments had failed in four different ways: a regex for a verdict format that matched nothing, hard-coded A/B labels, the wrong log source and an unpaginated fetch. Before trusting silence, verify the instrument against known-positive data. The audit script that followed paginated by construction.

The tools I ended up with follow the same reasoning. I hardened and integrated a local observability service that correlated requests, threads, turns and runs through a debug tap, with bounded in-memory retention, redaction and payload limits. In Ferrite, my own platform, an imported observability prototype used bounded buffers that dropped and counted rather than blocking application work. Its diagnostic bundle tied an incident to the triggering datapoints, metric windows and log entries with source coordinates. Counting dropped telemetry matters because otherwise a quiet log gives no sign that evidence was discarded.

What this changes is the form of the answer. “The agent failed” becomes a specific thread on a specific deployment and instance, a tool call whose handle shows the path it took, a pane or process in a known state, and a clear note about where the evidence stopped, which for those clones was a hypothesis. The green status at the top of that first run told you none of it.

Chris Chabot, November 2026