# The decisions a restart shouldn’t get to make twice

Author: Chris Chabot
Canonical URL: https://chabot.dev/blog/decisions-a-restart-should-not-make-twice/
Published: 2026-11-03

Which coordination decisions an agent runtime must persist before acting, which it can re-derive, and how recovery separates unstarted work from uncertain effects.

A single agent that restarts has a transcript to go on. It reloads its messages, gets told what was interrupted and carries on, and if it repeats a little work that is usually the worst of it. A group of agent tasks has a harder problem, because by the time its process dies the group has made decisions that don’t belong to any one transcript. It has accepted a plan, worked out which nodes are ready, reserved budget for provider calls already in flight, approved one node’s work and rejected another’s, and perhaps started writing an approved change into the shared tree. If those decisions exist only in the memory of the process that made them, the restart makes them again, possibly differently and without knowing what the first attempt already did.

This question kept coming up in the systems and prototypes I worked on during 2026. They took different positions on how much a runtime needs to remember at all.

## Keeping nothing of your own

The smallest answer is to persist nothing. [OpenAI’s Symphony specification](https://github.com/openai/symphony/blob/main/SPEC.md) takes that position on purpose: it asks for restart recovery driven by the tracker and the filesystem “without requiring a persistent database”, and it says plainly that exact in-memory scheduler state is not restored. The agent-written prototype I worked on that March held very little in memory: what was running and claimed, retry attempts, completions and token totals. All of that could disappear without much harm, because the orchestrator polled the tracker every 30 seconds by default, ran each candidate through its eligibility rules, and on a retry fetched the candidates again rather than trusting what it remembered about them.

That works because the decision that matters, whether this issue still needs doing, lives in the tracker, and keeping the tracker durable is somebody else’s job. That supplies the test I’d apply to everything that follows: if a decision can be re-derived from something authoritative, don’t keep your own copy of it.

## Decisions nobody else holds

A multi-role plan fails that test immediately, since nothing outside the runtime knows that a plan was accepted or that a reviewer passed a node. In two experimental runtimes I worked on that July, one centered on a journal and the other on a broker for file changes, those decisions had to be written down.

The journal runtime ran dependency-ready nodes in bounded waves and took each one through coding, a work receipt, an independent reviewer, approval and verification, with a combined acceptance check at the end. It kept all of this in a journal guarded by a reducer. Every event carried a sequence number, its own digest and its predecessor’s, and was checked against the run ID, generation, incarnation and fencing token, which is what stops a process that should be dead from writing into its replacement’s history. Transitions for the project and for each node came from explicit tables. Entering reviewing, approved or completed required evidence, entering failed or blocked required a reason, and a terminal project rejected further events.

Recovery follows from those rules. If the phase had moved past planning with no accepted plan, the runtime refused to continue. With an accepted plan it walked the nodes: anything ready, running, under review or approved went back to pending through a recovery event that preserved its evidence IDs, and when that node ran again it was told its earlier changes might already be present, while anything failed, blocked or cancelled stopped the resume outright instead of being quietly retried. So a restart couldn’t re-plan, couldn’t reverse a terminal outcome by accident, and repeated work caught in the middle with a warning attached, even when a node had been approved and hadn’t yet reached verification. Recording a decision and trusting it after a crash turn out to be separate choices, and this code was far more willing to do the first than the second.

Money needs the same treatment, and it’s easy to overlook because it takes effect at the provider rather than anywhere you can look. Both runtimes reserved budget before dispatching a call and settled the real figure afterwards. After an interruption, the journal runtime’s ledger sorted requests by how far they had gotten. A request that was reserved but never dispatched was a definite failure and could be treated as one. A request that had been dispatched was uncertain, because the provider may well have done the work and billed for it, so it was charged conservatively and then superseded. A provider response can still overshoot its admission allowance because input usage is known only after settlement. A reservation limits what you start, not what you end up paying.

The broker runtime took on a different question, which is who gets to write. Its workers didn’t touch the canonical tree. A worker produced an immutable proposal naming the path, the operation, the digest it expected to find there, the new content and a digest of the supporting evidence, and a single deterministic broker, holding a writer lease with a fencing token, applied proposals one at a time. It logged its intent, rejected the proposal if the expected digest had gone stale, staged the content, renamed it into place, read the file back and only then wrote a receipt. That order gave recovery something to work with. After a crash the broker could compare the file on disk with the digests in the pending intent: the old digest meant the write never happened, the new digest meant it did and only the receipt was missing, and anything else was reported as a partial integration conflict rather than guessed at. Its scheduler was just as wary, checkpointing nodes, evidence and receipt digests and budgets after each wave, returning running nodes to pending on restore, and pausing a node whose provider outcome was uncertain rather than pressing on.

Handoffs between tasks are a smaller case of the same thing. In the sub-agent code I consolidated in August, each child got a frozen brief and its own persisted session lane, and its full output stayed in the workspace as an artifact with only a bounded reference inline, so a long result could be read again later rather than surviving only as a truncated summary.

## Waiting for a person

An automation that stops to ask someone a question can’t use a browser tab as its memory, since the tab may be long gone by the time the answer arrives. In the server-side workflow runtime I worked on, human input was an explicit node in a workflow that could also call agents, run code and invoke MCP tools. When execution reached a human-input node, the run’s status became waiting, the current node IDs were stored and no finish time was set, so the run didn’t pretend to be complete. A resume was refused unless the run was actually waiting on a human-input node, the person’s selection picked the outgoing handle, and execution continued along that edge alone. Each wait had a timeout mode that routed to a timeout branch, continued or raised an error, a sweeper enforced those timeouts, and the resume could arrive at a webhook URL carrying a token, so nobody needed a session open.

One more state deserves a name, which is the store being unavailable. When the store couldn’t be read, the runtime raised a typed, retryable error whose meaning was that no workflow had executed yet, each workflow’s failures were isolated from the others, and past a marked point in the code that error could no longer be thrown. A caller could therefore tell “nothing ran, try again” apart from “something may have run”, which is the distinction every recovery path in this article keeps coming back to.

## Recovering after external effects

A durable record doesn’t buy exactly-once effects. An automation engine from March, written by an agent with me as co-author, wrote each claim to the run’s row, and after a restart it recovered runs that had gone stale for more than 30 minutes, marking them as recovered after a scheduler restart. If one of those runs had already called an external service, that call still happened. A runtime can know a run was interrupted without knowing how far its side effects got.

The coordination record is fairly short: which plan was accepted, on what evidence each node moved, what was spent or might have been, what reached the tree and how you know, who is allowed to write right now, and what you’re waiting for and until when. Whatever an outside system can answer, leave with that system. The rest has to be written down before you act on it, because the process that made the decision won’t be around to ask.