A running app still needs to publish a usable page

Macalicious is macOS Catalina rebuilt as a working desktop in the browser, developed with coding agents under my direction and structured as one application on Ferrite, the cloud platform I build on and operate. That application has three faces. API requests reach a pool of four workers through a queue, /health is answered by those same workers, and the page itself, the HTML, scripts, stylesheets and a few hundred pictures that make up the desktop, is served out of a blob by the platform’s gateway without any of my code running. This has real benefits, since a page load never touches the application’s process and a slow API call can’t hold up a stylesheet, but it also means that the thing answering the health check and the thing serving the page share almost nothing.

/health replies {"status":"ok"} as soon as a worker can take a request off the queue and publish an answer. It doesn’t look at the blob, and the first time it answers, the blob may not hold a page at all. The desktop’s integration test shows the gap in its own structure: it asserts that /health returns 200, and only then starts polling the blob, for up to a minute, for signs that the desktop has been published.

How the files get there

A Ferrite application has no build step and can’t read files when it’s compiled, because the admission gate refuses a build script and any include_bytes! of something that isn’t submitted source. The built files therefore have to arrive as source code. A packing script runs the Vite build and writes a generated Rust module with one base64 string literal per file, plus a build ID that is the SHA-256 of every path and its bytes. When the process starts, a background task decodes each file and writes it into the blob under its relative path, which is the key the content route serves.

So publishing the site is something the application does to itself on the way up, and every process carries a complete copy of the build it was compiled with. During a rolling deploy, processes from two different builds can all be doing this to the same blob at the same time.

Assets first, then the page

What makes that safe is a rule the packer enforces: every file except index.html must have a content hash in its name, and a build where one doesn’t is refused. Two builds therefore write different keys, or the same bytes under the same key, and index.html is the only path two builds can disagree about. Publication is arranged around that one path, with a small marker beside it that exists only once a publication has finished:

1. Delete the marker, ferrite-site.json.
2. Write every asset.
3. Write index.html, which makes the build live.
4. Write the marker, naming the build.

A page only ever references assets its own process has already written, so whichever build activated last, the page being served is one complete build. A process that dies partway leaves no marker, and the next boot of any build publishes again instead of trusting what it finds. Old assets are never deleted, so a browser still holding the previous page can load everything it asks for, at the cost of a blob that only grows.

A boot can skip all of this only when the marker names its build and the live index.html is byte for byte its own page, which costs two reads and keeps restarts and extra instances cheap. The second condition is there because the marker alone can be wrong. Two builds publishing at the same time can interleave so that the marker names one of them and the live page belongs to the other, and a later boot that trusted the marker would leave the wrong page live. The test for this swaps another build’s page into the blob and expects the next run to publish again, deletes the marker and expects the same, and then expects one more run to do nothing.

The content-hash rule reaches into packaging as well. A hosted submission has to fit the platform’s 16 MiB record limit, and the original 21 MB one was refused before a build even started, so the deployment build recompresses its pictures to 1024 pixels at quality 55. Recompressing a picture changes its bytes, so it has to change its name, which changes the text of every script and stylesheet that refers to it, which changes their names in turn. The packer settles all the new names before it rewrites any file, so that an import cycle or a dynamic import of a later chunk still ends up with the final name, and one name never holds two sets of bytes.

The job and its budget

That ordering takes care of what gets served. It says nothing about whether the job can finish, which depends on a number in the manifest. Every service in a Ferrite manifest declares a workload, and the site blob started out declared at 50 reads and 2 writes a second. A blob that only changes when you deploy doesn’t sound as though it needs more, and an earlier hosted deployment did publish the whole site with those numbers in place. But publication isn’t steady traffic. All of its writes arrive in the first seconds of a process’s life, from every replica that boots before a publication has finished.

The fix was two numbers: 512 reads and 128 writes a second, in units of 64 KiB, with a ten-second burst. That burst comes to 1,280 units, about 80 MiB, which leaves comfortable room for two replicas publishing a roughly 12 MiB site together.

What happens below that line is worth following through, because nothing about it looks like a failure from outside. When a write fails, publication returns an error, and the task that called it logs a warning, waits, doubling from 250 milliseconds up to a cap of 30 seconds, and starts again from the beginning. The new attempt deletes the marker, writes the first asset again and spends its fresh allowance on bytes that are already there, so a budget smaller than the job can stop publication at about the same place every time. Every write is idempotent, which is why the retries are safe, but safe isn’t the same as making progress, and a loop in that state never reaches step 3.

Meanwhile nothing else has noticed. The workers are running, /health says ok, the API answers, and the platform can truthfully report that the generation is deployed and a process is serving it. On a first deployment there is simply no page to load. On a later one the previous build’s index.html is still in place, so the site looks fine and doesn’t update. The in-process tests can’t see this either, since their harness is told which services exist and not how fast they may go, so the old numbers passed them as happily as the new ones.

What counts as published

Checking publication therefore takes two things /health can’t supply, the page and the marker, and the useful form of that check is the one a boot already makes:

GET /ferrite-site.json   exists, and names the build just deployed
GET /index.html          is that build's page, byte for byte

The two reads need each other. The marker is deleted first and written last, so a missing one means the last publication hasn’t finished, but a marker that’s present can’t vouch for the page by itself, for the reason above.

Those two establish that the build is published. The browser journey establishes that it works: it opens the desktop, checks that the page came from the blob the application published, launches Calculator from the Dock, types 12 + 3 = and reads 15 from the display, saves the result to the session’s tape through the API, reloads, and fails on any console error, page error or failed request. One passing run has exercised the blob, the assets, the scripts, the API route, the session cookie and the store.

What the order doesn’t settle

One limit remains, and a complete page makes it easy to miss. A content hash can tell you whether two builds are the same, but not which came first, and this application has no other notion of newer. An old process that restarts during a rolling deploy, after the new build has published, finds a marker that isn’t its own and does exactly what it was written to do: it publishes, and puts its older index.html back. Visitors get a complete but older page until the next boot of a newer process repairs it. Each process does log its deployment generation at boot, which might be the start of an ordering, but a generation counts control operations and not builds: scaling takes one too, and a process reports the generation that placed it. A rollback shows why newest would need defining at all, since there the older build is the one that should win.

So asset-before-index publication guarantees that what’s served is always one complete build, and it doesn’t promise the latest one. /health goes on answering the only question it was ever asked, which is whether a worker is alive.

Chris Chabot, November 2026