# The same scheduled occurrence, delivered twice

Author: Chris Chabot
Canonical URL: https://chabot.dev/blog/the-same-scheduled-occurrence-delivered-twice/
Published: 2026-11-03

Giving scheduled work a stable occurrence name so redelivery can be recognised, and what that name can’t settle once work leaves the commit or reaches a provider.

The applications on my platform had no timers. There was no resident polling loop and no ambient clock to wake them, so an application that wanted to do something every morning published a message with a future delivery time, and when that message arrived it published the next one. A recurring schedule was a chain of delayed messages, and any link in that chain could be delivered more than once, because a worker can die holding a leased message before it gets around to acknowledging it.

So the first thing a handler has to work out is whether the message in its hand is new work or old work coming back, and the delivery itself is a poor guide. The time of arrival changes with every attempt, so anything derived from the clock at delivery turns one firing into several. A message ID is steadier, but it identifies a message, and one firing can exist as two messages if the handler that published it crashed and ran again. What the handler needs is a name for the work itself: this schedule, due at this time, fixed when the occurrence is armed and carried by every copy. A nightly digest due at 06:00 on August 30 is the same occurrence however many times it turns up. The one due at 06:00 on August 31 is a new one.

Two systems I was integrating in August 2026 were built around that name: a durable workflow engine with its scheduler, and a pair of applications that ingest research sources and deliver scheduled briefs. Both had arrived in the repository from an earlier codebase, and my work was integration, repair and hardening. They did not share one implementation: the ingestion application's scheduling loop used a common consumer helper and its fetch loop was written by hand, the workflow engine had its own transitions, and delivery had a ledger. They are worth comparing because the idea is the same and the differences appear where the work leaves the system.

## The part that fits in one commit

The platform had a queue that could publish, delay, lease and acknowledge, and an atomic plane that could commit up to 64 keys in one envelope, optionally guarded by a key's current bytes. No transaction spanned the two. That leaves a short recipe. On each delivery, look for a marker under the occurrence's name; if it is absent, do the work and commit the marker together with the state the work changed; acknowledge last. A crash before the commit leaves none of that state or its marker behind, and redelivery runs the work again. A crash after the commit leaves a marker, and redelivery finds it and acknowledges.

Two copies running at once are settled by the guard. One commit lands, and the loser re-reads, finds the winner's marker and reports a duplicate. The scheduler's test suite has a case that races two firings of one occurrence and expects exactly that, one fired and one duplicate. Since the guard compares bytes, an admission that left a document unchanged would be invisible to it, so the schedule's chain document, like a workflow execution's snapshot, carries a sequence number that every commit bumps.

The name also protects the chain. With no transaction across the queue and the plane, arming the next link happens before the commit, and if the handler dies in between, the retry arms it again and the queue now holds two messages for tomorrow's occurrence. Nothing needs to be done about that, because both carry the same name and will meet the same marker. What the name cannot absorb is a second chain, which is why registering a source a second time was not allowed to re-arm it.

## Effects go first

Work that stays in the atomic plane is covered by that recipe. Work that leaves it is not, and here they part ways. The workflow engine sent every side effect before its commit, and its scheduler did the same when a firing started a workflow, published a message, queued a task or launched an agent. A crash in between repeats the effect, which the engine accepts openly: its documentation says the ordering exists so that each crash window lands on a documented seam. Because the effect can repeat, the name has to travel with it, so every effect a firing sent carried an id built from the schedule and the occurrence. One of the crash tests makes the consequence concrete. An effect goes out, the process dies before the commit, redelivery sends the effect again, and the consumer receives two copies under one idempotency key and applies them once.

The name did not prevent the duplicate. It made the duplicate recognizable, and only to a receiver that checks. [Temporal's documentation](https://docs.temporal.io/activities) recommends that Activities be idempotent so retries can be processed without duplicate effects.

The research ingestion application had the easiest version of this, because a fetch is a read. Its fetch loop ran a source's adapter and committed the resulting cards, the advanced cursor, the source's health, the knowledge events and the occurrence marker in one envelope. A crash in the middle of a fetch recorded nothing, and the retry fetched again from the old cursor. An occurrence that exhausted three attempts on infrastructure errors got a dead-letter record committed with its marker, so that redelivery stopped running it. Provider failures were handled separately through the source's health and deferral policy.

## Duplicate does not mean done

A marker says the commit happened. It says nothing about what was supposed to follow the commit. The ingestion application projected its cards into the other store after the envelope landed. The briefing application composed a brief, committed it frozen with its marker, and then enqueued deliveries, which is a step on another plane. A crash between that commit and enqueueing means redelivery hits the marker, and the deliveries may be missing. A handler that treated a duplicate as nothing to do would lose that brief without an error anywhere, since every later attempt would see the marker and stop.

So in both applications a duplicate was an instruction to re-drive whatever follows the commit. Ingestion re-projected its cards. Briefing re-enqueued the deliveries, which is safe only because deliveries have identities of their own, one level down. Freezing the brief belongs to the same argument. If composing ran again on a retry, one occurrence could produce two different documents and the name would be attached to something that changes. With the brief frozen in the commit, the occurrence has one document for good.

## Where the name runs out

Delivery is the hard end, because a message that has reached someone's phone cannot be recalled and not every provider will deduplicate one. The ledger, whose contract was ported from an earlier codebase, ran in a fixed order: gate, claim, provider call, settle. The claim was a guarded commit on its key being absent, so two deliverers got one winner. A settled delivery stopped there. A claim with no settlement was the ambiguous window, in which the send may or may not have left, and on a channel with no provider idempotency, which at the time meant Telegram, redelivery settled it as needing attention and never resent it. A transport failure, the same uncertainty arriving by a different route, settled the same way.

The code treated a 429 or a 5xx differently: it settled the attempt as failed and allowed a retry under a fresh key, up to four attempts, on a fixed ladder of sixty-second steps. That policy needs care, because a server error can arrive after an effect has happened. A fresh key is safe only when the provider's contract establishes that the attempt was refused or independently prevents duplicate effects. The decision to mint a new identity has to rest on what that provider guarantees about the failed attempt.

An ambiguous crash can leave a brief unsent until someone looks, which is the direction the ledger chose to fail in, a visible gap in preference to a duplicate it could not rule out. Where a provider does cooperate, the name can go further. The shared messaging code holds no state and runs no schedule, and its Resend builder simply takes the caller's idempotency key, which [Resend documents](https://resend.com/docs/dashboard/emails/idempotency-keys) as deduplicating for 24 hours. Even a receiver that checks has a finite memory for names.