A desktop can look correct and still lose your work

Macalicious, my rebuild of macOS Catalina as React components, had a bug that no screenshot could show. In Firefox you could open Notes, type a title and some text, click back at the end of it, carry on typing and press Undo, and that one Undo could take the whole note, title included. Firefox joins every run of typing that starts where the last one ended into a single Undo step, and clicking at that spot doesn’t end the run, nor does clicking elsewhere and coming back, nor focus leaving and returning. The window looked perfect before the Undo and perfect after it.

It is hard to fault the part of the project that checks how things look. A visual harness renders each example in a real browser at 1x and 2x and compares it with a crop of a reference screenshot from the 512 Pixels library of Catalina captures. Each comparison has two scores and a budget for each: a mismatch fraction from pixelmatch’s perceptual metric, which catches shape, and a mean colour difference, which catches a fill that is slightly off everywhere. An 8° hue shift on a push button’s bezel left the mismatch where it was and moved the mean from 0.26 to 2.00. The harness audits itself as well. A budget set too far above its measured score fails the run as slack, at the tight setting a corner radius of 4 pixels instead of 3.5 fails, and a specification that breaks the rules isn’t run at all, on the grounds that a comparison that cannot fail must not be reported as one that passed. For states no screenshot shows, such as a pressed button or a focus ring, it reaches the state with real input and checks chosen pixels against measured values.

One deployment’s report records all 5,550 of those comparisons passing, along with type checks, lint, every unit test file in Chromium and Firefox, the build and size gate and the Rust tests. A few days later the project was fixing the Undo bug, and nothing in that list had objected. A comparison captures one moment after a page has settled, and Undo is a sequence: a run of typing, a press, another run, a command, and then a question about what is left. In its tight tier the harness masks text out altogether, because off Apple platforms text is set in a fallback font and would dominate the score, so the words in the window are exactly what the strictest comparison is told to ignore. And the budgets belong to one engine. They are set in Chromium on Linux, where a push button’s bezels score 0.00 to 0.08 per cent, against 2 to 5 per cent for the same drawing in Firefox on the same machine. Of the desktop’s 28 comparisons, 21 came in above their Chromium budgets in WebKit and 18 in Firefox, from text rasterisation, antialiasing and colour management, none of which anyone would call a bug. Agreement with a screenshot tells you that a crop of the desktop looks like Catalina in that browser. It doesn’t tell you the desktop behaves like a Mac, and it tells you nothing about what Firefox will do with your typing.

Acting instead of looking

So the project has a second layer that acts instead of looking. End-to-end journeys drive the served desktop through Playwright with real input, and exit 0 only when every step passed and no page logged a console error, a page error, a failed request or an HTTP error. One of them walks the Edit menu in Notes, TextEdit and Stickies: paste a word, choose Undo from the menu bar and count the word, choose Redo and count it again. It then reads the document from the server, since text on the screen and text in the store are separate claims, and reloads the desktop to look a third time. That is a good journey, and it has its own blind spot. A paste is one discrete action, so the check says Undo works and says nothing about how typing is grouped. A behavioural test catches what its script happens to do, and losing a note takes a different script: type, click, type some more, change your mind.

The better question is in a census of the editors that gives every command its own row, driven by real mouse and keyboard input in any of four engines, with script in the page allowed only to read. The Undo row is careful about what it demands. Each engine groups typing into steps its own way, and macOS itself undoes “Typing” as one step, so requiring a particular grouping would test Firefox’s taste rather than the desktop. Instead the row writes a line, clicks after it with the real mouse and types a few more characters, and after Undo that last run must be gone, no word typed before it may be missing, the document must still exist, and Redo must bring the text back. A failing row counts the lost words and lists them. The assertion is about what a person can’t afford to lose, and it leaves the browser free to get there however it likes.

The fix stays out of the browser’s way for the same reason. A press of the pointer in an editable view marks the run as ended, and at the next typing key, if the insertion point sits at the end of a text node, it moves into a new empty text node just after it, which makes what is typed next a new Undo step. It does that only at the end of a node, because a split in the middle would move text that earlier Undo steps refer to, and never in a table, whose Undo the editors handle themselves. The history stays the browser’s own, so every route to Undo and Redo keeps working.

Ordering needs a test of its own, because neither a screenshot nor a script in one window can contain it. The conflict journey opens two windows of one session and edits the same note in both, holding back the first window’s save, as a slow network would, while the second window’s is stored. The held save then meets the newer revision and is refused, and the window must show the version saved first, say so, and keep its own. What makes the journey useful is how exactly it accounts for errors. Each application’s held save must be refused exactly once, that one 409 is excused, and any other console error, failed request or HTTP error fails the run. A test that ignored 409s would pass a system that refused every save, and one that forbade them could never run this scenario at all.

What could not be checked

The last layer is a report that stays honest about what it couldn’t check. The report behind those 5,550 comparisons says WebKit was not run, because it couldn’t launch on the host the gate ran on, which is a very different statement from WebKit passing. It records 50 of 51 live journey runs passing, all in Chromium, with TV at 28 of its 29 steps. The failing step is YouTube refusing to play the embedded video to that host’s network, with player error 150, which TV reports to the person as YouTube’s refusal. Nothing in the desktop is broken at that step, and the result stayed red anyway. The journey has a flag for this case and it is narrow: it accepts errors 150 and 101 only, only when TV shows its own explanation and offers Watch on YouTube, and any other error still fails the run. Census rows have a similar outlet, reporting a control as unavailable with its reason, which the row must find on screen, so that a browser refusing a menu access to the clipboard appears in the results by name.

Each layer fails for its own reason. The pixels fail when the drawing changes, the journeys when a command doesn’t do what it says, the readback when a change never reached the store and the conflict run when two windows race, and the report’s job is to say which of those ran, where, and what could not be checked. On a project built by directing agents, my job is largely to decide what counts as working, and I want those reasons kept apart. All 5,550 comparisons could pass beside a note that one keystroke could erase, and every one of them would be telling the truth about what it could see.

Chris Chabot, November 2026