# Shutdown can only stop what it can find

Author: Chris Chabot
Canonical URL: https://chabot.dev/blog/shutdown-can-only-stop-what-it-can-find/
Published: 2026-11-03

A sandbox coordinator fix that puts bootstrapping instances within reach of shutdown, and an API that says when it stopped waiting without knowing the outcome.

In the sandbox coordinator I was working on in the summer of 2026, starting a sandbox took a while, and most of that time was spent waiting. Once the provider had allocated an instance, the coordinator polled the daemon inside it up to twenty times at half-second intervals, started the terminal agent and waited for its WebSocket, uploaded crash-recovery state and waited for the WebSocket again, spawned the bridge to the workspace filesystem, and then gave a push-channel subscription up to 30 seconds to appear, after which it logged a warning and carried on. Those budgets alone could add up to the better part of a minute. For all of that time the instance was real, in the sense that the provider had allocated it and the coordinator was assembling local resources around it: a session manager, the filesystem bridge, a port forward.

A shutdown request that arrives during that window raises the question of what exactly there is to stop. The shutdown path worked from the coordinator’s map of its sandboxes, and a sandbox still bootstrapping could be missing from that map. Shutdown could therefore finish without ever hearing about the allocation, while the bootstrap carried on to its last step, which marks the sandbox Ready and records it, as though nobody had asked for anything.

The mistake is an easy one to make, because it’s natural to let one map answer two different questions: which sandboxes can I use, and which sandboxes am I responsible for? Using a sandbox depends on readiness. Shutting one down depends on ownership, and ownership begins at allocation, the moment something exists at the provider that somebody will eventually have to release.

The change I landed in that existing coordinator had two halves. The first registers the instance in the map as Provisioning before the bootstrap begins, which puts it within reach of the terminate path: mark the sandbox Terminating, tear down its local resources, ask the provider to terminate it, remove it from the map. The second half is a check at the far end of the bootstrap, just before it would mark anything Ready, asking whether the sandbox is still in the map. If it isn’t, shutdown got there first, so the bootstrap logs that the sandbox was terminated during bootstrap, removes the session manager, aborts the bridge, kills the port forward, asks the provider to terminate the instance and returns an error instead of Ready.

Both halves are needed because there are two independent parties here and each has to be able to see the other. Registration lets shutdown see the bootstrap, and the re-check lets the bootstrap see the shutdown. With only the first, the bootstrap’s last step would put a Ready sandbox back into the map after it had been shut down. With only the second, there would be no entry for shutdown to remove or for the check to consult.

## Observing cancellation

The re-check doesn’t wake the bootstrap in the middle of a wait. The bootstrap can learn about the shutdown when it reaches the check, or sooner if the instance disappearing makes one of its own steps fail. Threading a cancellation signal through every wait would respond faster, at the price of touching every step. The bounded waits limit the polling delay, but the check itself is not atomic with the transition to Ready. Because termination removes the entry only after cleanup, a still-present entry can mean that shutdown is already in progress.

The registration lives in the coordinator’s memory, so it gives a running coordinator a way to find allocations. If the coordinator dies during bootstrap, a provider’s own lifetime limit can supply an independent deadline for an abandoned instance.

Cleanup has an outcome of its own. These bootstrap failure branches called `let _ = provider.terminate(...)`, which for readers who don’t write Rust means make the call and throw away the result. The failure path could report that it had tried to release the allocation, but not whether the provider had actually released it.

## Telling the caller what you know

The same uncertainty has to reach whoever asked for the shutdown. I also bounded how long the gateway waits for a coordinator to delete an instance, so that the request could no longer wait indefinitely. The wait got a default timeout of 180 seconds. When it expires the gateway returns a 504, a transport failure returns a 502, and the message tells the caller that shutdown may still be in progress. The coordinator runs the shutdown as a detached task, so the gateway giving up doesn’t stop it, and the delete is idempotent, so the caller can safely ask again. A 504 here doesn’t mean the delete failed and it doesn’t mean the delete worked. It means the gateway stopped waiting, and the API says that rather than rounding it off into an answer.

A sandbox can exist before it is ready, and it can be stopping before it is gone. Those intermediate states need to be visible to both creation and shutdown, because that is where a slow provider or an interrupted bootstrap leaves the work. For anyone reading a start path, the practical version of this is a question to ask at every await: if a stop request arrived right now, what would find this work, and what would the work do when it found out?