Recovering a service deployment after its visible work has already happened

If a deployment fails before it has done anything, recovery is easy, because there’s nothing to put back. The artifact wouldn’t download or the instances wouldn’t start, and the controller can stop and say so. The failure that needs real design comes later, when the gateway has already been told to send traffic to the new revision and the controller then can’t write down that it did. At that moment the gateway says one thing and the catalog says another, and the rest of the system is about to act on the catalog.

Ferrite, the application platform I own, has a Cloud Run-shaped services layer on top of its core: immutable revisions, jobs, worker pools and a controller that reconciles them against what operators have asked for. A deploy is a saga with six steps, queue, install, instances, ready, routes and finalize, and each step’s completion is recorded durably, so that a controller killed at any point can resume where the records say. Routes come almost last because publishing them is the step everyone else can see. Finalize follows, marking the revision as serving and recording the new traffic on the service’s catalog document, and the space between those two steps is where the trouble is.

Most of what comes before the route can still be undone, so the controller tries to catch bad releases there. Readiness used to mean the first poll that saw a live process ID, which let a release that crashed a second after starting get promoted while the previous revision was drained. Instances now have to stay live for three seconds with no restart, and a test release that exits after a second and a half times out with traffic still on the old revision. No check of that kind helps when the store fails a moment after the route has gone out.

I direct this work with agents. The controller is operator tooling, proven against local fleets rather than carrying anyone’s production traffic, and the defects below came from a review that followed the code paths through exactly that window line by line, after which each fix got a test that forces the failure. The obvious recovery turned out to be wrong in a way that applies well beyond this controller.

“Done” describes the past

A route in this design doesn’t name a revision. It names a request group, which is a queue, and whichever instances are polling that queue answer what they lease, because a request envelope carries no revision at all. A new revision can be given a request group of its own and warm up beside the previous one, in which case moving the route moves the traffic. Otherwise the two share a group and the controller hands over, stopping the old revision before the new one starts while requests wait on the queue.

Take the first arrangement and let finalize fail after the routes step has published. The compensation used to be sensible as far as it went: stop the new revision and start the previous one polling again. It never touched the route, so the gateway went on sending every request to a group nobody was polling any more, and the service was down with its old revision healthy and waiting on a queue that no longer received anything.

Making the compensation publish the previous revision’s route again fixes that outage and exposes the second problem. The retry resumes the saga, and the platform’s saga helper runs the first step whose record isn’t done. The routes step is done, since it completed in the first attempt, so it gets skipped, and finalize records the new revision as serving while the gateway still points at the old one. An operator’s rollback during the retry’s backoff gets to the same place with no compensation involved at all. The saga’s record was accurate, the routes step had once succeeded, and it no longer described the world.

So finalize now publishes the plan it’s about to record, unless this attempt has already done so, and a resumed saga re-establishes the handover before it publishes or finalizes. A completed step’s record tells a resume how far it got. The resume still has to bring the visible state back into line with whatever it’s about to write down.

Whether to republish during compensation depends on what the failed attempt actually did, which the controller tracks with a flag set when publication succeeds. A failure before publication leaves the route table untouched, and a failure after it puts the previous revision’s routes back, or withdraws the service’s routes if there was no previous revision. A traffic verb such as a rollback has the same shape in a smaller space. It publishes and then records, and it restores even when the publish itself returned an error, because the publish may have landed before the error did.

Why the gateway and the catalog have to agree

All of this is worth the trouble because of what else reads the catalog. Garbage collection retires a revision once the catalog has stopped referencing it for ten seconds, and after a restart the controller rebuilds hosts for the revisions the catalog names. A rollback whose record failed used to leave the gateway on one revision and the catalog on another, so ten seconds later the collector would retire the revision actually carrying traffic, and a restart would dutifully repair the one that wasn’t.

The contract now is that after a failed deploy or traffic verb the gateway’s route and the catalog’s traffic name the same revision. The test that holds it there puts the two revisions on different request groups, so that a route naming the wrong group is a route nobody answers, and it relies on a few fault points compiled only into test builds, since no real input produces a store error immediately after a publish on demand. It fails finalize and asserts that the gateway is back on the old group, the catalog still names the old revision and that revision is the one answering pings, then retries and asserts the new revision in all three places. Then it fails the record of a rollback, checks that the route came back, and runs reconcile passes for twelve seconds, past the collector’s grace, to show that the serving revision survives them.

A retry has to recognise its own intent

Agreement after a failure gets you back to a consistent state, and the retry then has to continue the same deployment rather than begin another. The identity of an intent here is the digest of the revision’s configuration: a revision with the same digest whose saga never completed is this deployment, so the controller resumes it instead of minting a sibling. Two bugs showed how much that depends on its surroundings.

Finalize marks the revision as serving before it records the traffic, so a failed record left a revision marked as serving that had never served. The resume search only looked for pending revisions, which meant the retry minted a sibling and ran a second full saga. It now resumes any same-digest revision whose saga is unfinished, pending or serving. Separately, the garbage collector used to retire pending revisions after its ten-second grace while the reconciler waited thirty seconds before retrying a failed generation, so every retry found nothing to resume and minted another revision. Pending revisions now belong to their own collector, which keeps the one the desired state still wants.

The saga’s own identity has a similar trap in it. A deleted service can be created again, at which point its revision counter restarts and it reuses the same revision names. A saga ID built from the name alone would collide with the first life’s completed saga and resume it as already finished, without executing a step, which is exactly what the lifecycle test caught. So the ID carries the instant the revision was minted, which is stable across legitimate resumes and different for each life.

Deleting what you read

Operators describe what they want in desired-state documents, in the manner of a Kubernetes controller: the CLI writes a document with a generation number and a reconcile pass applies any generation the catalog hasn’t recorded as applied. Deletion is a tombstone in that same document, and tearing a service down takes a while, since it withdraws routes, stops instances and erases revisions and history before it removes the desired document at the end.

That removal used to be unconditional. A new generation written during the teardown was a perfectly valid compare-and-swap against the tombstone, it was acknowledged, and the last step of the teardown then erased it, leaving the service deleted and nothing anywhere to say otherwise. The delete is now bound to the version of the tombstone it read, so a newer document survives and the next pass applies it. The test pads the history with 500 entries to make the teardown long enough to write into, and lands its write once the history count starts falling.

The reconcile pass had one more way for a local failure to spread. It used to return at the first serving revision it couldn’t repair after a restart, before applying desired state, scaling, collection or schedules for any other service. It now carries on and reports that failure at the end of the pass.

Most of these bugs came from asking one record to do another’s job. The desired document says what someone wants, and can change at any moment, which is why cleanup has to be bound to the version it read. The saga’s record says how far one attempt got, which is what a resume needs and says nothing about whether that attempt’s effects still stand. The catalog says what the rest of the system should treat as true, so it has to be kept in line with what the gateway is really doing, because everything else believes it.

What the controller still can’t promise

One gap stays open by my own decision. On a request group shared between revisions, the handover stops the old revision before the new one polls, but the new revision starts polling before its own routes publish, so for that interval it can answer requests routed to the previous one. A revision with a group of its own doesn’t have that window. Closing it on shared groups needs a request queue per revision, and the token scoping for that design hasn’t been verified, so I’ve accepted it as a residual risk.

The other gap is plainer. The compensation is just another network call, and if it fails the controller can only log the failure and leave a retry to publish its plan, because nothing in the reconcile pass compares the route table with the catalog.

Job runs show what a queue of one’s own buys. By default they share one task queue, so a run records the queue’s published count before its first publish, and its settlement later scans from that count, which is how an envelope whose publish committed but whose reply was lost gets found and withdrawn before the next run’s workers start. With per-run queues turned on, each run gets a named queue and a worker grant scoped to it, and cleanup is deleting that queue, which can’t reach anyone else’s work. The tests for that mode send a controller process SIGKILL mid-run, and one deliberately broken version, in which every run of a job shared a queue name, passed the unit test written to catch it and took the process-kill test to expose. Cleaning up something you alone own needs far less knowledge of what might already have happened.

Chris Chabot, November 2026