Forgetting a memory must survive the retry that created it

The memory engine in Ferrite, the application platform I own, keeps what an agent should remember about a user as scoped records in a store, and everything else, the vector index, keyword recall and hybrid ranking, is derived from those records. Its service face is a pair of queues: a caller publishes a request, a worker applies it to the store and publishes a reply. Queues like this redeliver. A worker can crash between its write and its acknowledgement, a delayed copy can arrive behind a later write, and two replicas can each run one delivery of the same message, so any request the service has accepted may come back after the world has moved on.

Most writes can be made safe against that in familiar ways, and then someone asks the agent to forget something. Suppose a worker receives a request to remember that a user keeps bees, writes the record and dies before it records its answer. The user deletes the memory. Then the queue redelivers the original request, which still carries the text. There’s no longer a record at that key, so the create-only write that was supposed to make retries safe succeeds, and a memory someone asked to have forgotten is back, with nothing in the system aware that anything unusual happened.

A create-only write only guards against duplicates while the record exists, and deleting the record removes the evidence the guard was relying on. Anyone who has operated Cassandra knows a version of this: a delete is written as a tombstone, and if the tombstones are purged before a replica that missed the delete has been repaired, the deleted data comes back. In a system with retries, forgetting can’t be the absence of something. It has to be a fact in durable state that later writes run into.

I direct this engine’s development with agents, and we worked this out for the queue-facing service, with regression tests to hold it. The engine is proven locally rather than in production, so what follows is a contract and how it’s enforced, not a record of behaviour under load.

Giving the operation a name

A replay can only be recognised if the operation has a name that doesn’t depend on its delivery. On first arrival a mutation claims an operation record under its scope and a correlation ID chosen by the caller. The record holds a digest of the request, the ID of any memory it will create, fixed now so that every delivery targets the same record, and eventually its answer. Redelivering a finished operation returns that stored answer and changes nothing, and a correlation ID reused for a different request is refused rather than answered with another request’s result. The ID names an operation and authorises nothing, which is why the engine’s deployment rule gives each reply queue exactly one consumer app: callers find their replies by that ID, and anyone holding the queue could read the rest.

The gap that matters is a crash after the write and before the answer, because the operation record then says unfinished and only the memory record can settle whether the write happened. So the write carries its operation with it. A create puts its tag in the record it creates and an edit adds its tag under compare-and-set, in the same write as the change itself. One test corrects a memory to “reads in Spanish”, puts that operation back in the unfinished state a crash would leave, corrects the memory again to “reads in Italian” and then redelivers the first correction. It finds its tag, changes nothing, and the text stays Italian.

The first version of this kept a record’s last 32 tags and dropped the rest, which turns recognition into a window, and a delayed message has no reason to respect a window. Older tags now move to receipts under their own keys that never expire, and a tag leaves the record only after its receipt has been written, so a reader that doesn’t find the tag in the record it read will find the receipt. The test applies 80 more corrections after a create and an early correction, then redelivers both, and the text stays at “speaks 79 languages” with a single record in the store. The same reasoning covers the tags themselves. When their format moved from a 64-bit FNV hash to SHA-256, recognition had to accept both forms, because a message admitted before the upgrade can still be delivered after it.

Recognition also has to come first. A replayed supersede, which creates a record replacing an older one, used to check that its predecessor still existed before looking for itself. That predecessor may since have been forgotten, so an operation that had already succeeded could fail on a precondition it no longer needed.

A tombstone at the key

With those pieces in place, forgetting itself takes very little. A forget replaces the record, under compare-and-set, with a tombstone that keeps the ID and nothing it said: no text, no category, no entities, no embedding. Reads, listings and edits treat it as absent, and it’s never removed.

The key being taken does the rest. A create-only write fails there for every delivery, however they interleave with the forget, including a delivery that checked just before the forget ran. Because a record’s ID is fixed for one operation alone, a tombstone at that ID can only follow that operation’s own create, so a delivery that finds one treats its work as done and answers with the original ID. One test runs the sequence through a real queue and store, with a create, a simulated crash, a forget, a fresh worker and the redelivery, and asserts that the caller gets the same ID, a read finds nothing, a listing finds nothing even with superseded records included, and recall for “bees” comes back empty. Another races two deliveries of a create against its forget for 40 rounds.

This has a cost. Every forgotten memory leaves a key behind for good, every retired tag leaves a receipt, and nothing here prunes them, because the alternative is choosing a date after which a forgotten memory is allowed to return. It’s also worth being exact about what the tombstone promises, which is that the service won’t store, list or recall that text again. The delayed message holds the text until it’s acknowledged and the operation record keeps a digest of the request, so this isn’t a claim about every byte on every disk.

Corrections can be undone by accident

A memory can also disappear by being superseded: a new record names the one it replaces, and listings and recall hide the replaced record. That isn’t a flag on the hidden record. A record stays hidden only while a live record names it, which is a clean rule with a sharp edge. Consolidation collapses near-duplicates by keeping the newest in each cluster and forgetting the rest. If someone says “I live in Berlin”, then corrects it to “I live in Lisbon”, and later restates “I live in Lisbon”, the older Lisbon is the duplicate, and it’s also the one carrying the edge. Forgetting it would bring Berlin back without any write having said anything about Berlin.

So we changed consolidation to move the edge onto the kept record before forgetting the duplicate. A crash between the two leaves both edges in place, which still hides Berlin, and if the kept record already supersedes something else, it can only hold one edge, so the duplicate stays rather than lose it. The same rule means that deliberately forgetting a correction restores what it corrected, so a fact and its correction both have to be forgotten if both are to go.

A correct search over an old view

None of this is visible to a vector search, which computes a perfectly good cosine similarity over whatever it’s handed. The recall index is derived and cached per scope, so a record can be gone from the store and still score well. A worker updates its cache in place for its own writes, but a replayed write that changed nothing must not insert the record it found, since that record may have been superseded in the meantime. It drops the scope’s cached index instead, and the next recall rebuilds from the store. The test replays a create for “likes jazz” after it has been superseded by “likes blues” and checks that jazz stays out of recall.

Other writers are a different matter, because the cache only sees its own process’s writes. A record forgotten through another replica leaves this one’s recall when the cached index expires, ten seconds by default. So a forget takes effect at once in the store and within a stated bound in another worker’s recall, and a successful search can’t tell you which of those you’re looking at, since the similarity score is the same either way.

Chris Chabot, November 2026