Agent memory that remembers the current answer
Building a multi-tenant memory service where identity bounds access, later facts supersede earlier ones, and tests grade the final answer rather than retrieval.
In March, someone tells their agent that deploys go straight to production. Three weeks later they mention that deploys now go through staging, with a one-hour soak. In the middle of April they settle on a canary, five percent of traffic for thirty minutes. If they then open a new session and ask how deploys work, all three statements are about the right subject and all three sit close to the question in embedding space. Any retrieval system worth the name will find them, and only one of them is the answer.
That team is fictional. It comes from a benchmark I built in June 2026 alongside a memory service, whose job was to carry user and project facts from one agent conversation to the next across a multi-tenant platform. The scenario captures the half of the problem that a retrieval score tends to hide, and the other half is about who is asking. A service like this has to answer two questions that have very little to do with each other: whose fact is this, and is it still true?
Whose fact is it
The service was a Rust daemon built around an existing memory engine, embedded as a library. That engine comes from the lineage of the open-source mem0 project. Around it I built the control plane: tenant isolation, keys and grants, extraction and in-process embedding pipelines, consolidation, and the packaging that made the whole thing self-contained.
Isolation started with physical separation. Each tenant had its own vector collections, its own history database and its own directory on disk. Mandatory scope filters sat on top of that, dividing memory into user, project, thread and agent spaces, each reachable only with the corresponding ID.
The decision I think mattered most was about who gets to say who is asking. With service-key authentication enabled, trusted services such as our gateway held service keys. Agents held grants, short-lived bearer tokens that the gateway minted for one user and could narrow to a thread or project, with a default lifetime of fifteen minutes. A caller presenting a grant could not name a tenant or a user at all, because both came from the grant. Thread, project and agent IDs had to be absent or match the grant’s binding exactly, ingestion and administration were closed to grant holders, and every ID had to fit a strict pattern. An agent that had been talked into asking for somebody else’s memories had no parameter to ask with.
The path between conversations and the service mattered as much as the storage. I built on another contributor’s gateway integration, repairing its handling of privacy, identity and failure. Before a prompt went out, the gateway asked for relevant memories using the user’s grant. The service budgeted 300 ms for recall and the client gave up at 800, and on failure the turn went ahead with nothing injected, because optional memory should not be able to block the primary work. After a turn completed, the gateway posted the transcript itself using its service key, so agents never reported on their own behalf and arbitrary agent metadata was never trusted as identity. Failed turns and trivial exchanges were skipped, incognito sessions and memory-off settings opted out, and each ingested turn was recorded under a unique tenant, thread and turn, so a redelivered turn got back its original ID and status instead of creating a duplicate.
Memory is also a delayed injection channel, since text saved today gets read next week by a model with no memory of where it came from. Recalled material was therefore introduced as helpful context and never as instructions, with the current message taking precedence. An adversarial end-to-end suite went after the public API with credential-shaped saves, instruction-shaped saves, oversized facts and probes using foreign IDs, and planted an instruction-shaped memory for the consolidation sweep to clean up.
Is it still true
Keeping facts current started with how the store changed. The consolidation sweep, the only writer that acted without a user action behind it, compared memories and chose to add, update, delete or do nothing, and it skipped any ID the model had hallucinated. Recall then blended raw similarity with recency and a per-category prior, capped any one category at four results so that a single topic could not crowd out the rest, and rendered what remained into a token budget as a profile section and a context section.
In August the same ideas were reorganized for a different storage substrate in Ferrite, a separate project of mine. Records were the source of truth, semantic and keyword indexes built on standard HNSW and BM25 were derived from them and could be rebuilt, and a new record could name the one it superseded, which then dropped out of listings. That makes supersession a fact on the record rather than something a ranker has to guess at. One small test there shows the class of bug a scoped store invites: the key prefix for user a must not capture the keys of user ab.
Ranking and supersession still do not tell you whether an answer is right. The research had already pointed at this gap. LongMemEval treats knowledge updates and abstention as abilities to test in their own right, and the authors of LoCoMo reported that long-context and RAG approaches still lag well behind people, with long-range temporal and causal dynamics among the hard parts.
Testing for the current answer
The deploy scenario shows what a test for this has to do. The three statements go in through the real ingest API in order, with their dates, and the questions come later through the real recall API. A small model answers from whatever recall returned and a larger model judges the answer. Grading the answer is the point of the exercise, because a retrieval metric cannot see this failure: if the March statement and the April statement both land in the top five, recall at five is perfectly happy, and the agent may still describe a process that stopped existing weeks ago.
A small labeled proof tested retrieval alone: 15 queries over 24 memories, with a recall at five of 1.0, fourteen correct memories ranked first and one ranked second. It asked whether the expected memory appeared near the top of the results. The supersession scenario asks which of several related memories should determine the answer.
On LoCoMo, the same ingest, answer and judge setup got 980 of 1,540 questions right, about 64%: roughly 70% on single-hop questions, 68% on temporal ones and 45% on multi-hop. On the questions where the right move was to decline, it abstained correctly 385 times out of 446, about 86%. The 45% is the number I would look at first. A service can retrieve related text all day and still fail to put two facts together.
The deploy scenario itself belongs to a custom set of thirty-three questions across ten categories, along with corrected facts such as a team size revised from six to eight. The service got all thirty-three. Those questions give us regression cases for specific corrections and supersessions; LoCoMo adds the varied combinations that expose where the approach still breaks down.
For the deployment question, identity determines whose history can be searched, changes to the stored facts determine which instruction is current, and answer-level evaluation checks whether the agent uses it. All three statements can be relevant to the question; the memory service has to help the agent choose the one that applies now.
Chris Chabot, November 2026