# Finding the resource that actually stopped the build

Author: Chris Chabot
Canonical URL: https://chabot.dev/blog/finding-the-resource-that-stopped-the-build/
Published: 2026-11-03

A build died at the link step with most of its memory unused. A controlled comparison found the limit it had really hit, and the autoscaling work that followed needed proof of its own.

At the start of November 2026 a sample storefront called Shop failed to build on Ferrite, the application platform I build on and operate. It failed at the link step, on a machine with memory to burn: the job was allowed 48 GiB, its cgroup peaked at about 8 and it recorded no out-of-memory events at all. If you judge a failure like that by the size of the machine, you conclude that nothing could possibly have been short and you go looking for a bug in the toolchain or something odd in the Cargo setup, and we did try an experiment with Cargo, which made no difference.

A build on Ferrite doesn’t run against the machine, though. It runs inside a set of limits, each of which can stop it without reference to the others: a CPU allowance, a memory limit and a scratch filesystem for everything it writes while it works. At the time that scratch filesystem was 6 GiB, and the failing job had used 5.888 GiB of it.

A Rust build is good at finding that limit. It writes compiled artifacts for itself and every dependency into its target directory as it goes, and the linker then has to write the largest outputs of the whole job while nearly all of that is still on disk. So scratch use peaks late, and the step that runs out of room tends to be the last one, which is how a disk problem comes to look like a linker problem.

That figure of 5.888 GiB exists only because it was measured during the job. One of the lessons we wrote down afterwards is to measure a failing build while it is running, because a sample from an idle host can’t explain a job’s peak: by the time anyone looks at a builder after a failure, the job is gone and the machine looks as if it has room for anything. Even so, 5.888 out of 6 was treated as a hypothesis. It is suggestive, it is still below the limit, and a filesystem being nearly full when a linker dies doesn’t show that one caused the other.

I direct this work with agents, and the comparison we ran to settle it changed one thing. The same 94 source files and the same toolchain got a larger scratch filesystem and nothing else. Shop linked, passed its 538 native tests and reached 6.144 GiB. Acme, another sample storefront that had failed the same way, passed all 398 of its tests and reached 6.276 GiB. Those results do two separate jobs. The passes show that more scratch was enough, and the measured peaks show that the old limit wasn’t, because both builds went past it. The margin was only a few percent, and the failed run had left about 40 GiB of its memory allowance untouched.

The fix was a policy change to 12 GiB of scratch, which leaves both builds using about half of it, with memory, CPU and every other job limit left as they were. Ferrite’s upstream core already had incremental builder replacement, which retired the four old builders only as their replacements became ready, and app workers, core binaries, data and DNS were not touched. Shop and Acme then deployed the tested code at their existing per-replica budgets and passed focused browser journeys, and the JavaScript and CSS they serve match the bytes in the deployed source archive.

One number from that rollout needs keeping in its box. The prepared builder image we had selected was missing, so the new builders fell back to a native bootstrap that took about 17 minutes. That was how long capacity took to start, and it says nothing about how long an app takes to build, but a figure like that is easy to misfile once it lands in a table.

## What the fleet costs while nothing is building

The work that followed on November 2 looks related and is a different kind of claim. The scratch diagnosis was a controlled comparison of one failure. The fleet work was about how much machinery we keep running while nothing is happening, and its headline number is an estimate.

The baseline came down to 13 running machines: three control hosts, two for the edge and operator access, three data nodes, three application workers that can grow to six and two builders that can grow to four. Worker and data machines stayed large, since we had already learned in October what very small nodes did to ephemeral apps. Each build keeps its 28 CPUs, 48 GiB of memory and 12 GiB of scratch, and that is the design choice I care about: the saving comes out of the machinery that runs all the time and not out of what a job gets when it runs. The estimated steady cost is about 39 percent below the previous baseline. That figure is a calculation from published rates for those machines and their retained disks, before network, logging and other variable charges, so it is neither an invoice nor a cap. The same calculation puts two extra builders and three extra workers, running two hours a day, at a little under 4 percent more in machine charges, which is the argument for paying for bursts rather than for standing capacity.

## Watching the burst

That argument only holds if the burst happens, and a ceiling in a configuration file can’t tell you that. A growth limit says what the system is permitted to do. It doesn’t say that a new builder appears when four jobs arrive at once, how long that takes, whether the new machine can run a job or whether it goes away again. As it turned out, the declaration couldn’t even be trusted as a statement of permission: declaring a worker domain doesn’t by itself enforce the worker ceiling, so the durable zone cap has to be set and verified separately, and a healthy idle domain can report a maximum of three while its hard cap is six.

So we made it happen and watched. The autoscaling itself is native to the upstream core; what we chose was the baseline, the limits and the checks. One warm remote test took about 37 seconds. Four at once took between about 32 and 68 seconds and pushed the builders from two to four. Both new builders came up from the prepared job image, with their services running about 40 seconds after the provider started them and about 100 and 170 seconds after creation, against the 17 minutes it had taken with the image missing. Four concurrent deployments then took about 54 seconds each, job records showed both new builders had run successful jobs, and all four resulting apps answered GET, POST and repeated POST on their declared routes. Reclamation was observed as well: idle builders had already dropped from four to two before the burst, and afterwards supported drains removed the two older idle builders, which left the two hosts known to have the prepared image.

The provider’s own stock turned out to be a limit as well. Stockouts pushed the control and front hosts onto other machine families, which moved the estimate by less than half a percent, and changed the procedure: a host now needs a reservation for its target shape before it is stopped, because even its old shape can have gone by the time you want it back. On our test installation that morning the same kind of shortage had a simpler answer, since nothing official depended on it: I asked for whatever node sizes would get us moving.

Workers got a smaller proof. Raising their floor from three to four created and attested a new worker in about four minutes, and restoring the floor let a native drain retire an older one. Nobody pushed the fleet to six, so for workers that number is still a limit we have configured rather than one we have seen. Those short jobs also finished before collection ran, so this work produced no fresh peak memory or scratch figures, which is the same blind spot as sampling an idle host. The per-job limits were retained, and the number I would still quote for what a Shop build needs is the one from the controlled comparison: 6.144 GiB, in a filesystem that used to hold 6.