Testing a release without turning every release into a research project

Every time we went round the loop of installing Ferrite, verifying it and testing it, we found a few things we had missed. Learning that way is fine for a while, but by the end of October 2026 I wanted to get ahead of it, and I wanted two things that do not naturally go together: testing that caught more of what a user would actually trip over, and releases that did not lose half a day to testing.

Ferrite is an application platform whose core is developed upstream. On top of that core sit the services, sample apps and release tooling I own, work I direct with agents that write much of the code and run the release steps themselves. That last part shapes how the problem shows up. A capable agent handed a release will run a command, see what happens, inspect the server, discover that it needed to do something else first and carry on like that for as long as you let it. Each of those steps is reasonable. Added together they are a research project, paid for in time and tokens and repeated whenever the release changes, and at the end of it comes a broad test campaign, because nobody has said what could be left out.

So the first decision was that a release states what it has to establish before anything runs. The operator, human or agent, names the candidate, the baseline it is compared with, the contracts it affects, the live journeys it must pass and the conditions for being finished. The default scope is what changed. Full qualification is kept for a first baseline, an explicitly selected full release or a change whose impact is broad or unknown, and once it has been chosen its claims can’t be quietly reduced to fit a deadline. The full inventory of what Ferrite does still exists and is useful, but it describes the product. It isn’t a list of things to execute for every release.

Four layers of evidence

The scope selects from four layers. The first is the core, which upstream certifies. The second is the services we own, such as the dashboard and the AI service, whose essential workflows have to work before anything built on them means much. The third is a set of twelve primary sample apps, chosen for what they exercise rather than for their own sake: a book catalogue with more than two million records says something about bulk data and search, Macalicious about state that has to survive a reload, a ticker about server-sent events reaching a screen. The fourth is everything else, which gets tested when a change affects it.

Those layers only save time if old results can be carried forward, and the rule for that has to be more careful than “nothing changed”. A passing run applies to its exact artifacts and its exact installation, so the question to ask of any result is which of those two it depends on. A package test or an owner’s contract test is a fact about an artifact. A new binary invalidates it for that component and for whatever consumes it, and leaves everyone else’s results standing. A new installation works the other way around. In October Ferrite could not yet upgrade in place, so a new platform release meant a clean install, which feels as if it wipes out everything you knew. It doesn’t. The packages are the same packages and their evidence still holds. What nobody has established is whether credentials, isolation, capacity, DNS and routing work on these particular machines, so the runtime and integration checks have to run fresh, however green they were somewhere else.

The guide we maintain for this calls the rule reusing proof rather than claims, and it has some pleasant consequences. After a fix you rerun the failed or affected checks, not the campaign. If a summary has gone missing you recover it from the raw evidence, and you never rebuild an app to get a prettier report or a matching timestamp. If a reporting format can’t combine two valid scoped runs, that is a limitation of the format, and you keep the individual results instead of repeating journeys that already passed to keep it happy.

New coverage gets the same scrutiny. When we expanded the catalogue to 27 journeys and 57 expected outcomes on October 31, a fresh-context review recommended choosing added work by new capability or failure mechanism, and collecting measurements alongside that work instead of giving them campaigns of their own. Another instance of a failure we already check for adds run time and not much else. Some checks are cheap enough to be universal: during the rehearsal I’ll come to, all 45 published app origins passed 49 explicit HTTP expectations in under three seconds. A browser journey costs far more, so it has to earn its place.

A route instead of discovery

Discovery needed a different answer, which was to give the agent a route. On October 30 we added six focused skills covering cluster operations, demo operations, package verification, client verification, behaviour verification on the cloud and maintenance of the library itself. I took the approach from published verification skills, including poteto’s worked example and feature map and Cursor’s skills for creating and maintaining verification, and we deliberately left out their worktree and pull request creation, copied browser profiles and mandatory broad campaigns. Verification here means journeys and expected outcomes run against real installations. Formal verification stays available as a separately scoped option for invariants we own, rather than becoming another prerequisite for shipping.

The constraint I cared about most was that these skills own no execution of their own. They route through the existing installer, the existing runners and the existing evidence records, so there is no second installer or parallel ledger to drift out of step. A small read-only tool inspects those records without touching anything live and tells the next operator whether it is looking at new planning, a continuation, an uncertain attempt, legacy context or conflicting evidence. On November 2 I asked for a spring cleaning on top of that, noting that it was not spring, so that an agent could load a skill, read the instructions, execute a step, see that it did what was expected and go on to the next, dropping back to full reasoning only when something had changed or broken.

Rules for stopping

The part of this I would most like another team to steal is the rules for stopping, because a release with a clear scope can still grow from the inside. A release is a bad moment to discover that you would like a better test runner, so during one, code changes only to fix an observed blocker or something explicitly requested, and test infrastructure redesigns, extra fixtures, new generic runners and unrelated improvements all wait. The guide even fixes which model drives the browser, so that nobody opens a model comparison before the product tests. A broken check gets the smallest repair that works, and its original failure stays in the record, so that a mended check can’t quietly rewrite what the release did the first time.

Failures get a similar discipline, since each one is an invitation to do something expensive. Before anything changes, the failure has to be attributed to our code, the test, the provider or upstream, which stops an agent repairing the platform for what turns out to be a bad fixture. A retry needs new evidence and a stated limit, because a deterministic refusal comes back the same however many clean installs you pay for on the way to it. A wait needs a named completion condition, so that an agent isn’t spending turns inspecting state that hasn’t changed.

A red test then gets a decision of its own, because it records an unmet assertion and doesn’t by itself establish that the platform is broken. Some failures block whatever else is green: a failed security, account isolation, data integrity or certification requirement, a broken deployment or essential service workflow, or a named critical app, which is why a broken Macalicious desktop stops a release that depends on demonstrating it even when every other demo passes. Others call for an investigation within a stated bound, because missing measurements, timeouts or inconsistent observations could be hiding a shared fault. In Plan, one of the sample apps, the user’s actions succeeded while the telemetry never recorded its background loaders finishing. Those are separate findings: nobody gets to claim that the requests completed or that the cause is known, and it is escalated if it starts to affect visible behaviour, persistence, resource growth or other apps. An isolated defect in a noncritical demo, with the required user journey shown independently, can go out with the limitation, the decision and the follow-up written down, so that nobody reruns a release to clear one red line.

A written policy can also get ahead of its tooling. Ours puts full capacity proofs before their load tests rather than before basic service installation, while the installer went on refusing to pass its fleet phase without a complete builder receipt, and operators are told to leave that guard alone, since documentation alone doesn’t change what an executable does.

The rehearsal that did not finish

On the morning of November 2 I asked for a full dress rehearsal on a fresh test installation, watching for wrong instructions, any need for recovery or discovery, anything untested that should have been tested, skills that went unused or fell short, and delays. It found plenty, and what happened next is the useful part. A docs package pointed at the wrong installation’s origin and an optional sign-in secret was empty, among other faults. The repairs kept everything that had already worked: the docs were regenerated, the three missing apps deployed and the remaining domains bound without rebuilding their apps. The book catalogue was already active, so a read-only reconciliation confirmed its 2,360,648 books instead of uploading them again. When the provider ran out of the machines we had asked for, twice, the installation continued on other shapes and kept the hosts already running rather than starting again.

The result I like best from that day is a false one. The standard end-to-end capstone carries an implicit storage reservation that exceeded the write-rate budget of the mixed data machines we had ended up with, and it was refused. We selected a functional fixture that kept every upstream state, fork and isolation assertion with smaller declared workloads, and it passed all 39 assertions in about two minutes. The record then says that capacity qualification at the native standard remains false. Both statements are true, and a single green tick could not have carried them.

Plenty else passed, from real Google sign-in and session reload to provider calls with exact request and charge correlation and a fresh account taking an app through the claim lifecycle with its data intact, while browser claim submission, entry with no cookies and the full declared inventory of services and inputs were never verified. Later that day I asked for the installation to be taken down because it was still costing money, which ended the qualification where it stood. The unverified checks stayed unverified, the results describe an installation that no longer exists, and the next attempt starts on a new one.

When I first asked for this work I said I wanted to reach the point of being confident we were catching everything. The guide that came out of it says that no claim about a supported inventory guarantees the absence of every possible defect, and that when time or quota runs out you report the remaining blockers immediately and name every deferred check. That is a better sentence than the one I asked for, and on November 2 it was also the only honest way to finish.

Chris Chabot, November 2026