# What an agent can afford to lose

Author: Chris Chabot
Canonical URL: https://chabot.dev/blog/what-an-agent-can-afford-to-lose/
Published: 2026-11-03

Sorting an agent platform’s data into irreplaceable work and rebuildable cache, and letting that decide journal storage, migration, eviction and restores.

For a while our agents kept their session journals on the same shared filesystem as their workspaces. That filesystem is versioned and content-addressed, which suits source files very well, and the journal was a JSONL file that grew a line at a time as a session changed. This filesystem stored complete file versions rather than append deltas, so every line added to a journal that had grown to several gigabytes versioned the whole journal again. Each version of an append-only log is a prefix of the next, which means almost everything in that history was a copy of something already stored. On one long turn, measured on August 26, 2026, this came to about a gigabyte a minute.

Storage was already a live operational problem that week. A root volume had gone from 85 percent to full in under four hours, and turns failed once the disk was full. The journal was one pattern among those pressures, the one I pinned down in the days that followed, and fixing it raised a more useful question: when the runtime around an agent is replaceable and its work isn't, what exactly counts as the work?

## Moving the journal

I moved the journal out of the filesystem and into the agent's own durable state, replacing both the file on the shared mount and an interim journal that was local only. The session tree, meaning its entries, lane records and facts, was persisted as a flat, append-only log of mutations, with markers for creating and deleting a session. Pushing a mutation onto a tracked array recorded a single list-append delta, which the platform's persistence layer wrote into the instance's state database, the same database that held message history and that [Litestream](https://litestream.io) replicated off the host. Flushes were debounced, the platform consolidated deltas into checkpoints, and the host asked for a checkpoint at the end of every turn. Adding an entry now recorded one delta, and the new version of the entire file that used to follow every line was gone.

The operations in the log were exactly the mutations the in-memory store applied, and that choice is what made replay worth trusting. Opening a session rebuilt the tree through the same validation that produced it, checking that new ids were unused, that targets existed and that new lanes were valid, so there was one definition of a valid session and no second one hiding in the recovery code.

The journal used the platform runtime's existing delta persistence, database and replication. Its debounced flushes and turn checkpoints meant that appending an operation in memory was distinct from committing it to storage; recovery depended on what had been persisted before the host disappeared.

I had already made a similar change to the gateway's local state earlier that month. It kept its world as a single JSON document rewritten whole on each save; recorded timings put the cost on a 52 MB state at about 300 milliseconds per save. I replaced it with per-entity rows in SQLite and dirty tracking, so that a small update wrote what had changed. Mutations carried hints about what they touched, and a section with no hint was marked dirty as a whole and rewritten, which kept a safe path for small sections rather than demanding that every mutation be perfectly instrumented. The old JSON snapshot was kept under a timestamped name and the JSON backend stayed available as a way back. What a write costs should follow the size of the change, not the size of the world.

## Deleting the originals

The migration was the one step in moving the journal that destroyed anything, so it was built around being able to stop. Old journals lived in two locations, with journal blobs identified by their header bytes. The conversion was lenient about individual lines and strict about the result: the converted operations had to replay cleanly before anything was written to the database, and a journal that didn't replay was refused. Sessions with a live process and sessions already migrated were skipped, and a dry run was available. For everything else the order was write, checkpoint, prune, read back and compare, and only then delete. Read-back compared entry counts. That check could detect a count mismatch, but it did not compare the content of every entry.

A log that failed replay when a session was opened later was a different case, and it got different treatment. The migration could afford to refuse because the original was still sitting there. At open time, unusable log records were counted and dropped, while a session whose mutation replay failed was quarantined rather than exposed half-loaded. The host could then create a fresh session and rebuild conversation context from the gateway.

## What can be thrown away

Keeping the journal answers half of the question. The other half is which files can be evicted or cached at all, and the storage layout and runbook I reworked that August sorted this by kind.

Workspace content is work. It moved to the storage service's shared content-addressed pool with S3 behind it, and local objects could be evicted under disk pressure only once S3 had confirmed it held them, because evicting anything else would turn a disk problem into a data-loss problem. What the operations layer added was discipline around them: journals kept apart from versioned workspace files, hard rules that banned broad cleanup sweeps in favor of narrow, gated reclaim, a script for pruning old journal blobs that refused to run against a live process, and disruptive coordinator replacement that waited until turns were inactive.

Package and compiler caches are the opposite case, worth having and entirely rebuildable. The sandbox cache worked from an allow-list of rebuildable data, including Go, npm and pip caches, and defaulted to cold. Build trees, tool home directories, credentials and configuration were excluded, and that exclusion matters more than the list: a cache that restores them has become an implicit restore of secrets and authoritative state that nobody asked for. Older, broader archives were left untouched rather than silently reused. Each restore reported its outcome as off, disabled by selection, cold, restored or uncertain, and a missing receipt counted as uncertain and blocked the release of commands, since nobody could say what was on disk at that point.

Caches can carry authority in less obvious places. A separate image-cache implementation included a fingerprint of the registry credentials in its key, alongside the image tag, substrate hash and setup commands, so that a private image built with one set of credentials couldn't be reused by a request lacking them. The truncated SHA-256 was part of a lookup key; it stopped credential-mismatched reuse without encrypting image contents or providing a tenant boundary.

That sorting determined which data needed replication, which files could be evicted and what a cache was allowed to restore. For every file, the useful question was what happened if the host vanished, and whether the honest answer was "we rebuild it" or "somebody loses their work."