A prompt that holds still while the files change

Prompt caching rewards one behavior: sending the same bytes, in the same order, at the start of every request. The providers differ in the details. Anthropic caches automatically or at up to four breakpoints you set explicitly, and OpenAI caches a matching prefix by default and has added explicit breakpoints on newer models, but either way a change near the top of the prompt costs you the cached work behind it.

A coding agent is close to the worst possible customer for this arrangement. It reads files, rewrites them, runs commands that change other files and resumes from checkpoints, so the context it most needs to have right is also the context most likely to move. Keep every file it has read at the top of the prompt and refresh them as they change, and the cache never gets a chance to hit. Freeze them, and the agent reasons about code that no longer exists. Keep nothing resident, and it reads the same large files again and again and pays for them each time.

In August 2026, I built provider-specific replay and a resident file working set into an agent loop. The loop has since been retired; its design separated context that could stay fixed from changes the model needed to see immediately.

Replaying reasoning

Models that reason between tool calls hand that reasoning back in formats that belong to their provider. Anthropic’s thinking blocks carry signatures that have to travel with them, and OpenAI has reasoning items that a client can pass back across a tool call. Our loop had applied a single stripping policy to replayed history. On providers that would have accepted those earlier turns as they were, this invalidated a prefix for nothing.

So the policy became a property of the model rather than of the loop. OpenAI and xAI models replayed reasoning by default, Anthropic models and anything the code could not classify had it stripped, an explicit setting always won, and an invalid setting produced a marker rather than an exception halfway through a run.

The more instructive failure came from mixing providers. A conversation could carry a reasoning signature minted by one model family into a request bound for another, and our gateway’s Anthropic request builder trusted an empty provenance field on the block, which let the foreign signature through and earned a 400 in reply. The guard that followed removed the signature and its metadata together, kept a block that still held readable text, dropped a block that was nothing but a carrier for the signature, and treated an unknown family as unknown instead of as a mismatch. Telemetry recorded counts only, never the reasoning itself, and it measured what actually went out on the wire after the guard had run. A count taken from the configuration would have reported replay in exactly the cases where the guard had just removed it.

Keeping files resident

Source files were harder, because no provider rule settles the question. The aim was a resident working set: a block of files the agent had pinned, sitting near the top of the prompt, so that it could keep large sources in view without reading and paying for them over and over, while never mistaking an old copy for the current one.

The design split the state in two. Live pins moved freely during a turn as the agent read and edited. The block actually rendered into the prompt was the intersection of those pins with a snapshot frozen at an anchor, and anchors were few: the start of a turn, the commit of a condensation, a context recovery, an explicit refresh. For a given snapshot revision the block was byte-identical, and that one invariant is what let the design coexist with a prefix cache at all. A file pinned in the middle of a turn could not join the block without moving those bytes, so it rode in the uncached tail of the request, past the cache markers, until the next anchor absorbed it. The request looked roughly like this:

system prompt
resident files          frozen between anchors
conversation history    grows at the end
- - - - - - - - - - - - cache markers end here
additions               pinned or changed since the last anchor

The awkward case is an overwrite. Suppose a file is resident when the agent rewrites it. The resident copy is now wrong, but replacing it would invalidate everything cached behind it. So the old bytes stayed where they were, the new content went into the tail, and a note beside it told the agent that the copy there was the current one. For a while the prompt deliberately contained two versions of one file and a statement of which to believe.

Anchors did the reconciling. Each pinned path was checked against the workspace, unchanged content took a fast path, and changed content was re-read and reported as refreshed. A failed read was not treated as a deletion: the pin was marked unavailable and left pending revalidation, only a confirmed deletion removed it, and a separate guard covered the case where a large number of files appeared to vanish at once. One check exists because a long run turned up the need for it, rather than because anyone designed for it in advance. A read immediately after a sandbox write could return a truncated file while the write was still propagating through storage, and taken at face value that would have looked like a legitimate change.

The agent heard about all of this at the start of its next turn, in a note listing what had been refreshed, by someone else or by its own hand, and what had been removed, evicted, refused, narrowed, pruned or made unavailable. Anchors in the middle of a turn queued their notes for later. The model was told what had moved rather than left to work it out from the bytes.

Anchors also cost money. An anchor moved bytes, which threw away the cached prefix behind them, so a refresh in the middle of a turn had to earn that back. The loop gated it on expected payback and required a gap of at least two steps between anchors. Freshness had a price, and in the middle of a turn the loop weighed it before paying.

The block also had to stay small enough to be worth having. Its budget defaulted to a fifth of the model’s window, up to a fixed ceiling. A file over the per-file limit was narrowed to a slice of its head and tail, the least recently used pins were evicted first, pins created by an edit were never evicted automatically, and a file that could not fit was refused before anything was overwritten and returned inline, so the agent could still read it even though the block would not hold it.

A restart was the other moment when those bytes could have drifted. Checkpoints persisted identity alone, meaning paths, digests, ordering and snapshot metadata, and never file content. After a restart the next anchor re-read every pinned path and verified its digest, so a rebuilt working set rendered the same bytes for an unchanged workspace and a resumed agent got back the same block it had left.

Both halves of this work came down to deciding which bytes were allowed to move, and when. For reasoning, the answer depended on which provider would receive the request and whose signature was on the block. For files, it meant accepting that a stable prefix cannot be a faithful picture of the workspace at every moment, and that the model is then owed an explicit account of the difference: which copy is current, what changed and who changed it, what could not be read, and what was refused.

Chris Chabot, November 2026