# The test passed, but the person still could not do the thing

Author: Chris Chabot
Canonical URL: https://chabot.dev/blog/the-test-passed-but-the-person-could-not/
Published: 2026-11-03

What a browser test has to establish beyond a 200 or a successful click, followed along a new user’s path from documentation to clipboard to the app and its dashboard.

On November 1, 2026 I followed a link to Beats, a music player and one of the sample apps on Ferrite, the application platform I build on and operate. I got a Ferrite login screen that I didn’t think should have been there at all, clicked through it with Google anyway, landed on a sign-in dialog, pressed the button and watched the page reload to the same dialog. The day before, we had expanded our verification catalogue to 27 journeys and 57 expected outcomes, and none of it had told me about this. I asked for a fix, for another pass over all the apps looking for similar failures and for an account of how we had missed it, with the lessons going back into the journeys.

A deployment can succeed, a process can report healthy, a URL can return 200 and a click can land while the person on the other end gets nowhere. Those checks answer questions about the system, and they are worth asking. A test that stands in for a person has to answer a different question: after doing this, can they see what they came for, and is it still there after a reload? I direct this work with agents, which write the checks and drive the browsers, so a lot depends on what they are told to assert. It helps to follow the path a new user takes, because almost every step on it turned up a version of the same problem.

## The page returned 200

A new user starts in the documentation, which is where HTTP success tells you least. On October 31 we taught our existing HTTP runner to accept forbidden-content assertions, so that a check could require a page not to contain something, and gave it a local negative control: a 200 page containing the wrong deployment realm, which it has to reject. Pointed at the live docs, the new assertion took 462 milliseconds to catch setup, MCP and canonical links in the production documentation that led to our test installation. Those pages were being served perfectly well. Anyone following them would have set up against the wrong place. A tested repair went live, and the documentation’s browser check now visits Quickstart, checks its canonical, setup and MCP URLs and fetches both linked setup documents.

## The button took the click

I think that for most people the setup page is the only documentation they will ever read. They go to the page, press the copy button or the install link, and from then on they let their agent do everything. I said that had to be tested every way until Sunday and back again, and the replayable journey that came out of it covers home, Quickstart and Dashboard login at desktop and mobile widths. It checks the exact bytes on the clipboard, a fresh copy after each reload, the fallback when clipboard permission is denied, keyboard operation, the payloads inside the Claude and Codex links and whether someone who wants a custom setup can find out how to add an MCP server.

A deliberately delayed-script check then caught something the rest of that journey could not. The Copy control is rendered by the server, so it appears before the JavaScript that hydrates it has arrived, and it accepted clicks during that gap. On a fast connection the gap is usually too short for a test to land in. On a slow one a person presses Copy, nothing is copied, nothing says so, and they paste whatever was already on their clipboard into their coding agent. The repaired control stays disabled until its handler is ready. The hosted replay holds eight scripts back, observes the controls disabled, releases the scripts and then verifies the exact clipboard contents.

## The link opened

Past the clipboard the path leaves our product, and what we can prove gets narrower. Codex received both the short setup prompt and the full first-app instructions in an unsent draft, with its composer normalizing one blank paragraph separator and every instruction and URL intact. Claude Code’s official handler received the exact payload and stopped at its normal workspace-trust screen. We didn’t broaden any trust permission or send a prompt to get past it, so we have not seen what Claude’s draft looks like at that point. The public instructions now tell people to copy the prompt if the desktop link doesn’t open.

## The URL changed

Inside an app, it is tempting to assume that navigation has finished when the address has changed. In a client-side app the URL can update while the previous view is still on screen. Huddle, a sample document editor, has a local browser control that reproduces exactly that, a changed URL with the prior editor still present. A test that waits for the URL and starts typing can then write into the wrong document, which in a shared editor is damage rather than flakiness. So Huddle’s writes wait for the matching route, the active document, the exact title and the editor. The check on the way out got stricter too. A fresh-context review called the restoration check weak, and discarding and reloading now has to bring back the exact published content, with blank or unrelated text counted as a failure.

Flow, a kanban-style board, shows why that state has to be read back rather than assumed. When the server refuses a new card’s title, the person should still have what they typed, be able to correct it, see the stale error clear and be unable to submit twice. The old code failed that journey. The corrected one passed all five steps locally and on the hosted app, where the test also reloaded to confirm that the corrected card had been saved and then archived exactly the card it had created. The raw record of that run contains one HTTP error and its console diagnostic, the deliberate refusal, and they stay there. A clean console on that run would have meant the refusal never happened, so a rule that demanded one would be asserting the wrong thing.

The click is weak evidence in both directions. In a browser journey through our book catalogue a click timed out in the browser tool, immediate readback showed it had succeeded, and it wasn’t repeated. An early check on the Shop sample asserted a cart line before it had settled. The corrected driver waits for the rendered line, checks it after a reload and confirms its own cart is empty after cleanup.

## The row appeared

Some of what a person can observe isn’t on the screen they acted on. If I make a request, the dashboard’s request table should show it, under the right app. The correlator we wrote for that refuses to compare anything unless the installation, account, app and generation match on both sides. Each client request must then match exactly one dashboard row by method, path, status and time within a stated tolerance, and no match and two matches are both errors. The same review caught it accepting impossible chronology, which now fails. Its result carries its own limit in writing: these matches are not request-ID proof. In a live run against Macalicious, an ordinary GET returning 200 and a unique, deliberate GET returning 404 both appeared in the correct app’s table, and the 404 carries the weight, because nobody else had any reason to ask for that path.

The same pass produced a pair of numbers that look like a problem. Live requests measured 112.2 and 113.4 milliseconds at the client, and the dashboard reported gateway times of 6 and 7. The correlator’s result says in so many words that the difference is not a performance regression. One number is the whole lifetime the client observed and the other is the gateway’s view, and putting them in one column would invent a mystery.

## The smallest honest proof

Each of those checks comes with a negative control, because a check you have never seen fail hasn’t shown you it can: the wrong realm on a 200 page, the changed URL over the old editor, the withheld scripts, the deliberate refusal, the impossible timestamp. The same review recommended keeping one connected proof of action, readback, dashboard observation and cleanup, rather than piling up more isolated checks. It doesn’t have to be large. On the dashboard’s own secrets form, a temporary secret was created, observed after a reload, deleted and independently confirmed absent, and because secret values are write-only that establishes its name and lifecycle and says nothing about its bytes. In Macalicious, a note is created, read back from the backend, survives a reload and then exactly that fixture is deleted.

Exactly is the important word. Flow’s fix for renaming a list was only proved locally, because hosted columns are shared and a test has no way to create one of its own, and we were not going to rename someone’s shared column to get a green tick.

The pass over the apps confirmed 22 problems like these, and each got a repair and a journey that can replay it, most of them on the hosted apps as well as locally. One app, Reminders, couldn’t get that far, because after an account claimed the app the platform refused the next source commit under its former owner’s prefix, and we didn’t swap in another app or reset data to make the failure go away.

The question to put to a browser test is what it would have said while I was pressing Sign in and arriving back at the same dialog. If the answer is that it would have been green, it is testing something, and that thing isn’t whether the person could do what they came for.