When the older worker finishes last

Take a worker publishing a summary of some upstream change. It has checked that its summary is current, and it is writing the page body to object storage when fresh evidence arrives and a newer summary is accepted. A moment later the older worker’s write lands. R2 is doing exactly what it documents: a read after a write is strongly consistent and concurrent writes to one key go to the last writer. Nothing has failed, and the reader-facing page now shows a summary the system has already replaced.

That race is the sharpest version of a more general question about ingestion, which is how a job recovers from failure without lying about what it knows. A stale worker must not publish over newer evidence, a failed fetch must not move the cursor past data nobody read, and a source that couldn’t be reached must not look like a source with nothing to report. In an August 2026 change-tracking pipeline I built with coding agents, these questions came down to two places where work could be mistaken for progress: settling an ingestion cursor and publishing its summary.

One claim, one cursor move

Ingestion was built around a fence. Creating an ingest run admitted one active run per source by writing a claim token and the current cursor version in a single D1 batch, which is a transaction. A worker that outlived its lease had its run marked failed when a new run was created, and a durable workflow replaying its own run could re-enter it without waiting for the lease to expire, but only if its identity matched. Finishing required the token, and a stale token threw. The settlement itself was one more batch: the run’s row was updated only if the source still held this token and cursor version, the cursor advanced only if the run hadn’t failed, and an audit row went in alongside them, guarded so that there could be only one per run. A read-back then confirmed the settlement or threw. A stale worker that wakes up late therefore can’t move anything, and a failed run releases its claim and leaves the cursor where it found it, so the range it failed to read is still ahead of the cursor.

Summaries followed the same discipline. Accepting one required its evidence hash to match the event’s frozen evidence, and its identity was the event, the hash, the prompt version and the model, so repeating an acceptance with identical content was harmless and different content under the same identity threw. Accepting a newer summary retired the previous one in the same batch that inserted it. All of that is bookkeeping in D1 tables, and D1’s batch boundary covers it. The wiki page is a different case, because its body lives in R2, and D1 and R2 don’t share a transaction. Each store can make conditional decisions of its own, but neither can make one decision that covers both.

Across the gap

Publication first checked everything it could on the D1 side: the summary hadn’t been rejected, it was still current, the event still pointed at it and the evidence hash hadn’t changed. A summary that had already been superseded was refused at this point, before any card or object was written. If the checks passed, the sequence ran like this, with a newer summary arriving at the worst moment:

1. The worker upserts the source card, writes the page body to R2, reads it back and compares the body and its hash.
2. Meanwhile a newer summary is accepted, retiring the worker’s row and moving the event’s pointer in one D1 batch.
3. The worker tries to mark its summary as published. That update requires the row to be current and the event to still point at it with the same evidence hash, so it matches nothing, and the read-back throws.
4. The worker re-reads the event, finds the newer summary and republishes it onto the same page ID.
5. It rethrows, so its own attempt still ends as a failure.

The Miniflare test for this injects the supersession inside the emulated R2 write itself. The old publish rejects, the page body is the new summary, the old row is unpublished and no longer current, and the winner is published and current.

Two choices in that sequence are worth defending. The loser repairs the page instead of merely failing, because a worker that only reports its defeat leaves readers looking at the summary it has just written. And the repair goes one level only, with recursion explicitly switched off, so publishers can’t chase each other back and forth across the gap. The cost is a little mess, since the stale source card stays behind, attached to nothing.

The repair is one more write across the same gap, so it can fail, and a worker that dies between losing the update and republishing never runs it at all. Cursor settlement happens once per claim token, the audit row once per run, and the page converges only if that one repair succeeds.

A missing result is not a zero

If an attempt declares results that it never admits, a later cursor position does not recover those missing results. The rewrite that followed made spent retry budgets explicit: when a window’s retry budget was spent, or its final worker never came back, the attempt went terminal rather than being claimed or offered for ever. That gave an operator something to act on instead of letting later progress hide the unfinished work.

That distinction runs through all of this work. An empty result from a source I reached is a fact about the source, an empty result from a source I couldn’t reach is a fact about my pipeline, and the job of recovery code is to keep the two apart. An older worker that finishes last loses the update deciding what is current, and a failed fetch cannot advance its cursor as though it had read the missing data.

Chris Chabot, November 2026