# Making an agent’s memory useful without slowing its next reply Author: Chris Chabot Canonical URL: https://chabot.dev/blog/agent-memory-without-slowing-the-next-reply/ Published: 2026-11-03 Capture, evidence, retrieval, projection and delivery can each fail alone. A small replay found a fact lost before search could see it, and 92 live calls carried no receipt that any memory had arrived. The goal for our coding agents' memory was easy to state. A later conversation or project worker should find the relevant earlier lesson, understand when and where it applies, follow its evidence back to the source, and avoid treating an expired observation as current truth. It also had to do all of this without making anyone wait, because memory is optional and the reply is the thing the person actually asked for. That sounds like a search problem, but search is only one stage of five. Something has to capture the right evidence, keep its source and validity attached to it, retrieve it when it is relevant, project it cheaply into the request, and then deliver it to the agent that is actually running. Each of those stages can fail without the others noticing. Within about a week we had a ranking that found everything in a small test, a stored copy that could no longer find one of those things, and a sample of 92 live calls with no receipt to show that any memory had arrived. Most of what follows was implemented and tested locally, with its new features off by default, and the last section is about what happened when it had to reach running agents. ## One home for each kind of fact The first decision was about authority. Learned facts, preferences, procedures and lessons live in a versioned memory file for their scope. Goals and standing instructions stay in the project brief, owner decisions stay in their decision records and outstanding promises stay in the coordinator's notes, and retrieval can point at any of those without copying them into memory. The search index sits beside all of this as a disposable accelerator. Its terms, aliases and postings can nominate a candidate but are never enough to return a fact: every candidate is read again from its current canonical version, with privacy, lifecycle and applicability checked at that point, so a stale index has no way to bring back something that has since been retired or forgotten. I also decided against two things memory systems tend to accumulate. The new memory is built from existing sessions instead of an imported corpus, a legacy format and a migration programme, and there is no human approval queue. Agents extract, reconcile and publish memory themselves, and an inferred synthesis can be used as soon as it exists. The safeguard is labeling rather than gatekeeping: metadata records whether a person said something, an agent inferred it or a tool observed it, and no label waits on anybody's sign-off. A label grants nothing either. Inferred text cannot promote itself into an owner instruction, and neither a high retrieval score nor an inferred relationship creates permission to act. ## Telling the model what it is looking at The first correction was small. Our prompt renderer was leaving out temporal and lifecycle metadata that the store already held, so a model could be handed an expired fact with no way of knowing it had expired. Each entry now gets an evidence view, computed when it is read and never written back: who the source was, whether this is a statement, a synthesis or an observation, where it is in its lifecycle, whether it is inside its validity window, and whether it applies to the repository, environment and version under discussion. These are separate questions, so an entry can be disputed and expired at once without either status hiding the other. A source labeled verified is describing its own history and does not make the assertion true forever. An expired fact is labeled as history rather than deleted, a disagreement is shown with its counterevidence rather than settled by picking the newest timestamp, and an old owner instruction stays in force, because an evidence clock does not cancel an instruction. Time makes this awkward. A validity interval can run out while the file's version stays exactly the same, so a cache keyed on the version would go on serving a fact past its expiry. Keying on the clock would be worse, since it would rewrite a stable prompt on every step and lose the prompt cache along with it. So the projection's identity includes a validity epoch. When a turn is admitted we record the time and work out the earliest future boundary among the entries in play, the projection is reused until then, and a cheap in-memory comparison at the next step boundary notices when it has passed. Only the affected part is recomputed, and it reaches the model as an ordinary context update, with no polling of files and no interruption of a request in flight. ## Finding it Retrieval starts with what can be explained. Exact matches on entry IDs, repository paths, issues and work items come first, and identifiers keep their punctuation and repository context, because `#42` in two repositories is not one entity. Lexical candidates then go through a deterministic BM25-style scorer, drawn fairly across authorized partitions so that one large repository cannot take every slot. An alias can nominate a candidate without proving identity, and an ambiguous one returns the alternatives. We had no embeddings to build on, since the memory paths we inspected had no embeddings API, and I treated one as a later substitution that would have to earn its place in measurement rather than as a prerequisite. Semantic recall began as something cheaper: an explicit ranked lookup with weak lexical coverage may spend one metered model call on at most three short search variants. That output is treated as untrusted search input. It cannot change scope, validity, permission or evidence, it cannot displace an exact match, and a generated phrase never appears as a discovered fact. If the call fails, the lexical results stand and the response says its semantic coverage was degraded, and the whole mechanism stays off until it beats lexical recall in a paired evaluation. Automatic recall, the kind nobody asked for, makes no new model calls at all. Better retrieval was not allowed to replace what already worked. Threads and coordinators still receive their complete current memory and workers keep their standing entries, rather than being reduced to whatever a top-k query returns. An automatic supplement can only add current entries that a worker's selection left out, up to 2,000 tokens, and it is omitted rather than allowed to evict anything. That supplement also stays disabled until it demonstrates a benefit. ## The miss that ranking could not fix The most useful result came from a very small test. We replayed 40 excerpts copied from three development threads, all from one episode of work, with a deterministic stand-in playing the extraction model, so the run exercised the production capture and storage guards and said nothing about how well a real model extracts. Then we asked four questions with frozen source labels at each stage. Literal lookup found none of the four, ranking over the raw excerpts found all four, and ranking over persisted memory after a restart found three. The fourth was not a retrieval failure. The production guard that keeps transient content out of memory had dropped one of the 40 excerpts, and it happened to be one the questions depended on, so no amount of work on the ranker could ever have found it. A score computed only on the raw excerpts would have reported a perfect result for a fact the system no longer held. Forgetting the first target on purpose then took the score to two of four, which is what forgetting should do, and we kept the denominator at four throughout, because relabeling the test would have hidden the loss. Four questions from one episode are not a benchmark, and the acceptance gate we set for this work is 80 independent episodes. What the replay gave us was the location of a specific loss. It is not enough evidence to change the guard, so that decision waits for the broader evaluation. ## Getting out of the way My condition for turning any of this on was that memories could be created, checked and retrieved without slowing anything down. Capture was the obvious problem, since it could sit in the foreground of preparing a prompt. We moved ordinary capture to a durable SQLite queue beside the gateway's state, with two workers and one active job per owner. Running the real hook with two stand-in extractors of 300 ms each, foreground preparation took 609.76 ms with inline capture and 0.49 ms with the queue, which made no foreground extraction calls and no writes. That measures the dependency we removed, and it says nothing about a provider's time to first output. Moving capture into the background opens a gap, and the gap needed closing. Source material still waiting in the queue stays available in the existing context envelope, so a fact does not drop out of the agent's view while it waits for extraction. The queue resumes its jobs after a restart and repairs terminal callbacks it missed, and in the test it eventually saved both pending activities with the exact final cursor. Background capture also let us stop clipping long sources, which had been discarding the middle of long messages: coverage is now tracked durably, a source is published only after it has been extracted completely, and incomplete extraction stays visible as incomplete. Reading got the same treatment. Requests draw on an immutable local projection built by a background publisher, so preparing a request should involve no network calls owned by memory, and a missing or unverifiable projection is simply omitted. Warm search over generated records measured a p95 of 3.06 ms at 10,000 records, not counting validation against the canonical service, and a 100,000-record index was refused by its capacity bound and reported partial coverage, which is the behavior I want from a bound. The first version of index preparation stalled the event loop, so it now yields in bounded slices and builds in worker batches. The labels are the cost that remains: a generated 1,000-entry file carries about 53 KiB of them, so annotation has a real price at size even when retrieval adds no I/O. We do not get to pay that price by dropping standing instructions or mandatory context to improve the latency score. ## Agents manage it too A fast projection covers what an agent should know before it starts. It does not help when the agent realizes a memory is wrong, or when someone asks what happened last time. I wanted agents to have search, list, read, edit, retire, forget, audit and bounded consolidation tools alongside the projection, and to know that they have them. In my experience agents are quite capable of this, and one that sees an invalid memory should be able to remove it. Owner confirmation remains as an optional correction and never as a required review, and a remembered permission still does not authorize anything. When personal agents needed memory as well, I asked for them to use this engine rather than get a second one, since two systems would mean doing the latency and relevance work twice. The personal-agent work already used the same document, store and file APIs, so most of what remained was composition: account-private, thread, project and repository scopes in one background publisher and one projection, each bound to its source and privacy versions. The hard case there is forgetting, because an account-level forget has to reach every process and work scope that received the fact, including across restarts and partitions. ## Whether it arrived Everything so far stops short of the last stage, which is publishing the projection to a running agent, and that stage was still failing once the code reached our development host. One project acknowledged a repaired publication and then failed its renewals, another failed its recovery outright, and in a 12-minute sample across both, 92 completed agent calls carried no memory receipts. Capture of historical material was still advancing, but the freshness of current sources and reliable delivery were both still open, and ordinary automatic memory stayed off by default. That sample shows why the last stage needs its own evidence. Optional memory must never hold an ordinary reply, and I would not change that rule, but it means a delivery failure produces an agent that answers on time and remembers nothing. Those 92 calls say nothing about the overhead of memory, because nothing shows that any memory was in them, however good their latency looks. Local preparation and the first visible response have to be measured separately, and the evidence that a memory reached a running agent is a receipt on the call that carried it.