# Keeping a coordinator responsive while its workers are busy

Author: Chris Chabot
Canonical URL: https://chabot.dev/blog/keeping-a-coordinator-responsive-while-its-workers-are-busy/
Published: 2026-11-03

When a coordinator shares a process with its workers, load and every recovery action land on both. How the deciding agent was separated from the process that holds the tree, sandboxes and workers, what the relay between them costs.

In October I asked why our project coordinator used a sandbox at all. It doesn't compile anything or run any code. It talks with the person who owns the project, calls models, and works through its workers and the file and GitHub functions the SDK provides. The answer was that the coordinator doesn't use the sandbox. The process it runs in does, because that process was also hosting every one of its workers.

In our cloud system at that point, a project ran as one instance hosting one Bun process. That process holds the project's versioned file tree, creates the sandboxes as leaves of that tree, and runs every worker and sub-agent as a lane, our word for an agent session inside a shared process, all on one event loop. With workers and sub-agents together that can be a hundred or more agents at once. This has real benefits for resource use and coordination: messages between lanes are in-memory calls, checking and claiming a file is one synchronous operation, a parked lane costs next to nothing, and a wait-for graph across every agent is free to inspect. The difficulty was that the coordinator's own turns ran in the same process, so the agent whose job is to decide shared a fate with everything it was directing, and it did so in two ways that have little to do with how hard coordination is.

The first is load. Everything on the event loop waits for everything else, and on 19 October one project's host reached a 2.8 second event-loop p99 at 4.2 GB, at which point the coordinator's turns, and the admission gate in front of them, slowed down with its workers. The fixes from those days show some of what was competing for that loop: per-lane scans that accounted for 40% of the process's CPU, state rotation running on the event loop, 13 GB of state in the runtime that manages checkpoints and resource limits, a 34 GB session store, and a checkpoint step in which one stalled read held every lane's turn because all the lanes shared that runtime. I wouldn't name a dominant cost from that list, since that's a question for a profiler and it changes with the workload. What they have in common is that their blast radius is set by how many lanes share the process.

The second is lifecycle. The platform's recovery acts on processes and instances, and it has no idea that one process is playing two roles, so every action taken for one role lands on the other. A rollover, which moves a runtime onto a new release, suspended every lane and re-attached every sandbox whenever it moved the coordinator. Restarting a wedged coordinator killed every worker. A stop that couldn't be confirmed for one worker escalated to restarting the coordinator, which meant all of the workers. The coupling ran the other way as well: an idle coordinator was being woken just to host a lane of finished work, and a worker's message to an idle coordinator was delivered only into that process's memory.

## Sorting the fixes

To see how much of this came from the pairing, I had coding agents go through the roughly 700 commits made since 17 October and pull out the ones concerning how lanes are hosted, which left eleven families of fixes. Each fix got one of three marks: it exists only because workers share the coordinator's process, it would still be needed by any process that hosts lanes, or it has nothing to do with the pairing. I'd recommend this to anyone who suspects a structural problem and can only see its symptoms, because it turns a feeling into a list. Some fixes were there purely because of the pairing, such as a run of repairs to a retired-lane check that failed resumed worker turns after every coordinator move, which only mattered because every worker session sat in the coordinator's one session store. Many others would be needed by any host, but the sharing multiplied their consequences. And no commit in that window touched host health gating, recycling, rollover itself, the listener that starts lanes or the supervisor for stuck runtimes. The mechanisms we might have suspected hadn't changed. The breakage had come through the pairing, each family of fixes was a local defence against one coupling, and new couplings would keep appearing for as long as one thread played two roles.

## What can actually move

Named separately, the two roles are simple. The coordinator is a conversation that decides: it calls models and gateway tools and runs no commands. The lane host is a process that holds the tree, creates and keeps the sandboxes, and runs every worker. The obvious fix is to get the workers away from the coordinator, and that runs straight into the real reason for the pairing. The workspace store allows one writer per tree, and only the instance that owns a workspace can create its sandboxes, so the tree, the sandboxes and the lanes that write to the tree have to stay together on one instance. That store is shared infrastructure owned by our platform team, and changing it wasn't on the table. What can leave is the coordinator, whose only uses of the shared process were its read tools on the tree, its tools for listing and messaging workers held in memory, and its environment tools for the shared sandbox. Everything else it did already went through the gateway.

So the first phase moves nothing that is running. A project that opts in keeps its old coordinator thread as its lane host and gets a new thread for the coordinator, whose process holds no sandbox, no tree and no lanes. The control record gains a field naming the host, which for a project that never separated is simply the coordinator's own thread, and every place in the gateway that used to read "the coordinator" had to say what it meant: the instance holding the tree and the lanes, or the agent that decides. Turns, wakes, actions, decisions and status mean the agent. Sandboxes, the tree, project state and the lanes mean the host.

Because this is a re-designation rather than a migration, the only hard question is when. The dispatcher takes it at a quiet point: no coordinator turn is open, no stop is in progress, no turn's actions are still waiting to be applied, and no wake has been claimed or is waiting on memory admission. It finds or creates the new thread under a deterministic effect ID built from the project and the host, and then one control commit names the host, names the new coordinator, bumps the coordinator's generation, resets its delivered conversation context and adds an inbox entry to wake it. The tree, the sandboxes and the running lanes stay where they are, and the worker threads, whose metadata already named the old thread, don't need to know. One of the tests separates a project with a running worker and checks that the worker is untouched and that exactly one thread was created. Rejoining is the same commit in the other direction. The price is paid once per project: the new conversation starts from the project's record, its memory file and its snapshot, so anything the old conversation knew and never wrote down is gone.

Afterwards each runtime is judged by its own work. A rollover holds the coordinator's runtime only for its turn, its wakes and its unread inbox, and with no lanes to suspend and no sandbox to re-attach, moving it is cheap. Restarting the coordinator releases only the coordinator. Admission and host health gate worker starts on the host, while a coordinator wake is held by the coordinator's own runtime, so a slow host no longer delays its turns. A worker start that finds the host down asks runtime recovery to bring it back, at most every 15 seconds per host, rather than waking the coordinator to do it, and a sandbox grant for the coordinator's thread is refused outright.

## The relay

The coordinator still needs thirteen tools that act on the tree and the lanes, and those now live in another process. They keep their names and parameters. Underneath, the toolkit sends each call to a gateway method, which checks that the caller is the project's running coordinator turn and that the project is separated, then passes a control message down the host's event socket, the same exchange the system already used to ensure a sandbox exists. The host runs the named tool outside any turn, for at most 55 seconds, and answers with the tool's own result. A host whose stream has closed is brought back once and asked again. Anything else comes back to the coordinator as an ordinary tool error.

The bound creates the problem every bounded call has, which is that a call the host didn't answer in time may still have run. Reads don't care and can simply be repeated. A message to a worker is a different matter, because a worker that receives an instruction twice may act on it twice, so the relay makes one attempt, never repeats a message blindly, and the tool error tells the coordinator to check the worker list before it sends again. Messages travelling the other way changed status too. A worker's message to a separated coordinator lands in the host's memory, which the coordinator no longer shares, so the durable project inbox that had been added days earlier to fix a narrow bug with idle coordinators became the only way such a message could arrive. What had looked like a patch turned out to be load-bearing.

## What it costs

Separation makes the coordinator's deciding independent of the host without doing the same for its reading. Every relayed call is a trip over an event socket, which adds latency and some load, and those reads still wait on the host's event loop, so a host at a 2.8 second p99 makes them slow even though it no longer touches the coordinator's model calls, admission or turns. Reading files through the platform's files API directly from the gateway would remove that dependency and is the first follow-up. The rest of the bill is smaller: another instance per separated project, though one with no sandbox; a host that lives on lane demand and can be released when idle, so that a start after a release has to wait for it to come back; and two bindings carrying the coordinator role so that the lane machinery could stay unchanged, which means code added later could mistake the host for the coordinator, although grant checks compare the caller with the coordinator's thread and active turn, so the host still can't act as one.

Version skew needs care as well. An older agent build on the new coordinator thread would ignore the new configuration, get a sandbox and read an empty workspace of its own, and an older gateway would take the new coordinator thread for the host and find no lanes there, so the agent release has to be live before anyone turns the setting on, and every separated project has to rejoin before a rollback that far.

Separation also does nothing for the host itself. The workers still share a session store and that same runtime, and one worker's overload is still every other worker's problem. The design for that is three kinds of process per project: a coordinator that decides and holds nothing, a tree custodian that owns the workspace and creates every sandbox but runs no lanes, and one or more worker hosts that each run a bounded number of lanes, so that a failure, a move or an overload takes down only its own. What stands in the way is file access, because a lane's file tools use a mount that only the tree's owner has, so a worker host on another instance would have to route them through the sandbox the lane already works in. That phase is not built.

The first phase was built behind a per-project setting, off by default, and the one measurement taken at that point was of its cost to the dispatcher: over 200 passes on a settled project it added no storage calls with live state on, at about a tenth of a millisecond per pass before and after. That shows the placement check is cheap to carry. It doesn't show that separation makes a coordinator more responsive, which would take a live comparison on one project of relay latency, the host's event-loop p99 and the coordinator's turn latency, before and after. I'd want that comparison before building anything else, because I would rather find out what one relay round trip costs on a real project than design three kinds of process around a guess.