# The invalidation that arrives while the loader is running Author: Chris Chabot Canonical URL: https://chabot.dev/blog/the-invalidation-that-arrives-while-the-loader-is-running/ Published: 2026-11-03 Why a cached entry must record tag epochs before its loader runs, why writers invalidate after they write, and why leases only limit duplicate loads. Suppose one request misses the cache and starts loading a product page from the database. A second request updates that product and invalidates the tag the page depends on. If invalidating means deleting entries, it finds nothing to delete, because the entry it ought to kill does not exist yet. The loader then returns the old row and writes it to the cache, where it is served with every appearance of health until it expires. Nothing has failed, nothing has been logged, and the only evidence is a customer looking at the wrong price. I had to settle this while building the data layer of an application framework. Its replicas ran independently, each with an in-process cache in front of a shared durable one, so the slow loader and the invalidation could easily be on different machines. The answer came down to two ordering decisions, one in the reader and one in the writer. Limiting duplicate loads is a separate job, and neither decision relies on it. The framework took its vocabulary from Next.js, borrowing the names of [`cacheLife`](https://nextjs.org/docs/app/api-reference/functions/cacheLife) and [`revalidateTag`](https://nextjs.org/docs/app/api-reference/functions/revalidateTag), and its stale window from [RFC 5861](https://www.rfc-editor.org/rfc/rfc5861). Underneath, invalidation never touches an entry. Each tag has an integer epoch in a separate atomic key-value plane, and invalidating a tag advances its epoch by one. A cached entry carries a witness, the list of tags it depends on and the epoch each one had, and a read fetches the current epochs and compares. Any difference and the entry is dead, whatever its TTL says. This changes the shape of the race, because an invalidation is now a fact recorded against the tag, and that fact can condemn an entry that has not been written yet. What remains is a narrower question: when should a new entry read the epochs it records? ## Before the load or after it There are two obvious moments, before the loader runs or after it, and after is tempting because that is when the entry is being built and you would like its metadata to be current. Here are the two requests in that hypothetical schedule, with the tag starting at epoch 7. ```text Request A (cache miss) Request B (update) read epoch of "products" -> 7 loader reads the old row write the new row advance "products" -> 8 loader returns write entry: old row, witness 7 ``` The next reader fetches the current epoch, gets 8, compares it with the witness's 7 and recomputes. Now move A's epoch read below the loader. The entry becomes the old row with witness 8, the next reader compares 8 with 8, and the old row is served until it expires. The advance has been recorded as something the entry already knew about. The entry is born stale, and nothing left in the system has any way to notice. Reading first can still be wrong, but only in one direction. If A's loader happens to see the new row just before B's advance, the entry is correct and its witness fails anyway, and somebody pays for a recompute they did not need. A false alarm costs a database query. A false all-clear costs an unknown stretch of wrong answers. So the loader read the epochs first, and in fact read them only once, at the top of the request, where the single read served the in-process check, the durable check and the witness of any fresh entry. The safe order also turned out to be the one with the fewest round trips. That one read settles a few other things. The in-process cache was validated against the same epochs as everything else, so correctness never depended on it: a replica did not need to be told about an invalidation in order to honor it, because every read asked. An in-process hit saved the fetch of the entry and never the check. The comparison ran before any time-based logic, in the order witness, expiry, staleness, which kept the stale window from bringing an invalidated entry back. Staleness is a tolerance you opted into when you chose a cache lifetime, but a failed witness means someone has said this data is wrong. And because the comparison was for equality, an epoch that had moved backward also killed the entry, as it should after a table reset. ## The writer has an order too Look again at Request B, because it holds a second ordering that matters just as much. B writes its row before it advances the tag. Swap those and the witness is helpless: A can read the new epoch, load the old row before B's write lands, and store that old row under a perfectly current witness. So mutations write first and invalidate second, and since the durable store and the atomic plane commit independently, the gap between those two steps is real. One of the framework's reference applications, a social feed, shows it plainly. Its repost toggle wrote to the store with a stamp identifying the action inside the written value, then committed the tag advance. A crash between the two leaves new data in the store and old pages in the cache until the action is retried or the cached entries expire. The retry finds its own stamp, knows its write landed and goes on to advance the tags. The stamp makes the store write recoverable on retry; it does not make the write and the invalidation one transaction. The advance itself is atomic within limits. The atomic plane committed envelopes of at most 64 keys, and it had no increment and no prefix scan, so advancing meant reading the current epochs, adding one and committing, guarded by a single version key. That guard occupied one of the 64 slots, which left 63 for tags, and the framework refused a witness with more. An advance that kept losing the guard gave up after eight attempts and reported a conflict. An action could put its idempotency marker in that same envelope, leaving at most 62 slots for tags before any other effects. Recording the action and invalidating its tags were then one bounded atomic commit. ## Stopping the herd is a different job None of this stops a popular key from being loaded by every replica at once. The authors of [Scaling Memcache at Facebook](https://www.usenix.org/conference/nsdi13/technical-sessions/presentation/nishtala) name stale sets and thundering herds as the two problems their leases address. In this design those are separate mechanisms: the witness handles stale sets, and leases handle only the herd. There were two leases, because an entry that is absent and being filled and an entry that is present but stale and being refreshed are different states with different holders, and one key for both would let a refresh still in flight suppress the fill of an entry that has since expired. Each was a create-if-absent write with an expiry, ten seconds for a fill and thirty for a refresh, values chosen rather than tuned. An expired lease reads as absent, so a holder that dies stops blocking later attempts once its lease expires. A healthy holder deleted the lease on every exit including failure, because a lease left behind by a failed load would send every other reader down the re-read path to find nothing, and a broken upstream would look like a cache that had stopped filling. A reader that lost the refresh race got the stale body at once while the winner awaited its loader inline, so this was not the background revalidation RFC 5861 describes: everyone else was served immediately and one request paid. A reader that lost the fill race re-read once, shared the winner's entry if it had landed, and otherwise ran the loader itself. So these leases limit duplicates and do nothing stronger. A fill can still run alongside another, as that last clause allows, and the lease value was a constant and the release an unconditional delete, with no ownership token and no fence, so a load that outlived its lease could delete its successor's lease on the way out. I would not claim one loader per key, and the design does not need that claim. Entry writes were guarded on the version the reader saw, losing that race was not an error, and whatever got written still carried epochs from before its load. A lease failure costs extra database queries, and the question of whether an entry can be believed stays with the witness. ## The direction of a mistake There was another ordering problem in the witness check. A witness sorts its pairs by tag so that its encoding is canonical, but the validity check compares those pairs by position against epochs fetched in the caller's order. A caller that lists its tags out of order can get a miss when the reordered epochs differ. As long as the tag set stays the same and epochs only move forward, a misaligned comparison cannot pass after a tag has advanced, so this mismatch can make a call site stop caching without making an invalidated entry pass. It needs fixing, but it is the right kind of bug to have. The regression test I would add starts with a loader that advances its own tag before it returns, followed by a read that must recompute. It exercises the ordering directly: move the epoch read below the loader, and the second read accepts an entry that should have been invalidated.