# Keeping scheduled work moving after a crash

Author: Chris Chabot
Canonical URL: https://chabot.dev/blog/keeping-scheduled-work-moving-after-a-crash/
Published: 2026-11-03

A morning pipeline built on durable, date-keyed obligations, leases and a five-minute sweep, so a crash between steps cannot silently drop the work that is owed.

In August 2026 I built an unattended morning pipeline with coding agents for my personal assistant system: gather research, compose a briefing from it, publish and send that briefing, write a jobs report and update a wiki. Any of those steps could die partway through, which is ordinary and the sort of thing retries exist for. The awkward place to die was between two steps. When the research succeeded, something had to start the composition, and the natural way to write that is a callback in the same process. A crash destroys a callback. The research is durably recorded as succeeded, I am owed a briefing and a publication, nothing anywhere is going to produce them, and because no component failed, no component has anything to report.

That failure is silent by construction, and it set the shape of everything else. If a process can vanish at any moment, then what the pipeline knows has to live somewhere that outlasts the process, and the thing most worth recording is what is still owed. So I built the morning around durable obligations: a named piece of work for a particular day, with a record of how far it had got and what had to happen next.

## Naming the work before doing it

The first thing that has to survive is the identity of the work. An obligation's key included its owner, product and local date, with the date worked out from a durable profile rather than from the server clock or whatever a caller happened to pass in. That matters because [Cloudflare's cron triggers run on UTC](https://developers.cloudflare.com/workers/configuration/cron-triggers/) and mornings don't. The whole chain was driven by a sweep on a five-minute cron, and during that sweep a planner declared each day's work. In one of the planner's tests, a sweep at 23:20 UTC on 16 August is already twenty past midnight on the 17th for an owner on British Summer Time. The planner declares the new day's obligation, since the day has started, and queues nothing, since the morning window hasn't opened. A sweep at 04:05 UTC queues it.

Because the key is derived rather than issued, creating an obligation is safe to repeat. The insert does nothing when the key already exists and raises a conflict only when the new request carries a different intent, so the automatic planner and a manual dispatch converge on one obligation instead of racing to create two. A changed profile deliberately isn't a conflict. The profile version is recorded as provenance and kept out of the identity, so a later sweep restamps an obligation that hasn't started, which beats raising an error nobody can resolve. An owner with no durable profile gets a refusal with the gap named, because without a profile there is no local date to key by, and I would rather see a named gap than a guessed date.

## Steps that remember their effects

A worker claimed an obligation with a single conditional update that moved it to running, incremented its attempt count, used the new attempt number as a lease token and set the lease's expiry. Completing the work was fenced by that token and an unexpired lease, and the continuation, the row recording what came next, went into the same batch as the completion and its audit entry, under the same guard. [D1 treats a batch as a transaction](https://developers.cloudflare.com/d1/worker-api/d1-database/), so the batch is the boundary the design leans on. Leases and fencing tokens are standard distributed-systems practice, and I wanted their consequence: a worker that has lost its lease can't complete anything, and a completion commits together with the plan for what follows.

The morning's continuation was a declared sequence of four steps: compose the brief, publish it, produce the jobs report and enhance the wiki. A later step was refused until the earlier one had completed. Before performing any effect, a dispatcher wrote an action row under its lease, with an idempotency key derived from the obligation and the step, and when the effect came back it had to report both the provider and the provider's own reference for what it had done. Uniqueness is enforced on that pair, because two unrelated providers can legitimately return the same message ID, and a globally unique reference column would refuse the second one after its side effect had already happened, which is about the worst moment to discover a schema mistake.

Suppose a dispatcher starts the publish step, writes its action row, calls the provider and disappears. Its lease expires, the next sweep finds a step in flight with a dead owner, and every instinct says to retry. The service refuses, because a second call against a provider that doesn't deduplicate would perform the effect twice. The in-flight action first has to be abandoned with evidence, which means naming the token that was observed and showing that the owner's lease is dead, and the replacement then submits under the stored idempotency key rather than a new one.

That ceremony buys less than it may appear to, and it helps to be exact about how much. A dead lease shows that the previous owner lost its claim. It can't show that the process stopped, and the original request may still be on the wire, so protection against a double send ultimately comes from the provider or the execution ledger honouring the shared key. Recovery also costs money. A continuation has a limit of five attempts, which stops a step that keeps dying from retrying for ever, and each of those attempts can be expensive: if a dispatcher dies after the model has written the briefing but before the step is recorded, the replacement can compose it all over again, and I pay for that even when the publish further down the chain is deduplicated. The design protects the effect a reader would notice twice and accepts paying again for one that only the invoice notices.

## What actually moves it along

Advancing the plan had two mechanisms sharing one implementation. A fast path could run as soon as a workflow completed, and a sweep ran on a timer, reading only the database and advancing anything the fast path never reached: a continuation whose callback was lost, whose dispatcher died mid-plan, or that was never created at all. A succeeded obligation's budget for sitting unadvanced was 15 minutes and the cron ran every five. The cadence was declared in the code as well as in the deployment configuration and asserted on every invocation, because a sweep that runs less often than its budget can't meet it, however correct the sweep is.

The deployed chain relied on the five-minute sweep. A completion hook could record a failure and return because the sweep could recover the same work whether a callback ran, failed or was lost. The morning chain ran from a cron handler and rows in D1, so its progress remained explicit application state. Whether I have been sent my briefing is a fact about my application, and no platform's guarantee about its own steps could settle it for me.

The sweep's running order shows its priorities. It expired what had run out of attempts, ran its escalations, recovered stalled actions, advanced continuations and recovered outbox leases, and only after that did it plan the morning and poll its sources. A healing process has nothing to heal on a morning nobody declared, so the planner's job was to make sure the obligation existed before any run was dispatched.

## Four things that look like done

The last thing that has to survive is the difference between states a lazy status field would merge. A succeeded obligation says that a worker holding a valid lease finished its work. A published brief says that the edition's evidence was validated again, its reader copy was audited again and its bundle was stored as published, at which point it completed a separate obligation of its own with the bundle as output. In the outbox, a provider accepting a request and a destination being observed to receive it are different states, and the second has to come from a named mechanism. A provider saying yes tells you about the provider.

The read-only observer behind my morning canary was built on those differences. It read the durable profile, derived the day's key, required a succeeded obligation with an output hash, loaded the bundle and checked that it was the right kind and actually published, read the outbox row and required the audio artifact to exist. A timestamp it couldn't parse was a refusal. A missing piece made it throw with the missing thing named, so a failed check carried a real reason and it would not report a ready time the evidence couldn't support.

What the obligation model gave me was a way to ask, on any morning, what was owed, how far it had got and which piece of evidence was missing, and to get the answer from the database rather than from a process that may no longer exist.