# A catalogue import is not finished when the upload ends

Author: Chris Chabot
Canonical URL: https://chabot.dev/blog/a-catalogue-import-is-not-finished-when-the-upload-ends/
Published: 2026-11-03

A large book import passes through a source fixture, uploaded objects, an activated generation and a reader's filtered view, and each has its own count. Why a check has to name which count it means, and how to verify one without uploading again.

An agent checking my book catalogue demo followed a perfectly ordinary path through it: search for a title, open the result, go back, reload. The query and the results survived the reload, the detail page showed the right title and authors, and everything a reader would notice was correct. The catalogue it was reading held 52 books, all of them synthetic, while the description of that journey still talked about the million-book import.

The demo is a port of Vercel Labs’ [next-books](https://github.com/vercel-labs/next-books) onto Ferrite, the platform I build on and operate, with URL-driven filters, title search, pagination and detail pages, and it is normally filled from the UCSD Goodreads book graph. A gap that size between what a check expected and what it found can mean the product lost its data, or an import never finished, or the check is expecting something nobody promised it. The first two are incidents. On October 30, when this happened, it was the third: the journey’s default was a deliberate leftover from an older data set, the catalogue then live was a small synthetic one, and the app was behaving correctly. Getting the original data set back became a separate task that I had asked for. The check got a new requirement, which is that it has to be bound to the fixture the importer actually loaded, and a mismatch with an old default no longer counts as a product failure.

That fixes one check. The underlying mistake is to talk about “the catalogue” as though it were one thing with one size.

## Four things that can each be finished

A fresh deployment of this app contains no books at all. The catalogue isn’t seeded at deploy. It arrives through a host-side importer talking to a separate import face on the app, which is the one part that needs a credential, since the catalogue itself is public to read. What arrives is stored as immutable blocks and indexes, and each reader’s personal reading state lives apart from them, so an import never has to reason about anybody’s reading.

That arrangement gives an import four separate stages, and each can be complete while the next hasn’t started. The source data set can be prepared and byte-verified, as the full 2,360,648-book fixture once was, with nothing uploaded. The objects can be uploaded, every block and index in place, while no reader can see any of them. A generation can then be activated, which is the publication marker: until that happens a new instance says so in its own words, with “No books yet” in the sidebar, “No books have been imported” on the shelf and a count of zero. And finally there is what a reader gets from a query, which depends on the activated generation, the indexes and whatever filters the view applies.

I’m fond of that empty state. The upstream project has a four-book preview fixture and the port never shows it. A catalogue that falls back on sample data when nothing has been imported looks healthier than one that says it is empty, and it has just hidden the one fact you needed.

When the upload ends, then, you have learned about the second of those four things.

## A green suite on a limited run

The sandbox run on September 14 shows how far apart they can get. We deployed the app through the released CLI to an isolated fleet and started the full import, which stopped at 5,384 of 10,971 objects when the platform’s named-queue node ran out of memory. That is an object count, and it would be a mistake to read it as about half of the books. What went live instead was a generation labelled as a limited run: the first 400,000 archive rows, 233,954 browse rows and all 829,529 authors. The repository’s browser suites and 35 recorded real-browser journeys passed against that deployment.

Those journeys were real evidence that the application worked and no evidence at all that 2.36 million books could be imported, and the label on the generation is what kept those two statements apart. A search-and-open journey passes against a quarter of a million browse rows, and in October one passed against 52 books. It can’t tell you which catalogue it was reading unless someone has told it what to expect.

## Which count?

By November the project had accumulated a number of true figures for the size of this catalogue, and they are worth setting side by side.

| Figure | What it counts | When and where |
| --- | --- | --- |
| About 1.38 million | Browsable books with covers and metadata | The app’s README, an older description |
| 2,360,648 | Books in the full import fixture | Prepared fixture, reconciled on November 2 |
| 10,971 | Objects in the full import | The September run, which reached 5,384 |
| 233,954 | Browse rows in a limited generation of 400,000 archive rows | The September run |
| 1,058,671 | Results under the default filters | Reconciled on November 2 |
| 52 | Synthetic entries | The live demo on October 30 |

None of these is wrong. They come from different snapshots and different scopes, and you can’t get from one to another by subtraction or take a percentage across rows. A check that says the catalogue has the right number of books has not said anything until it names its noun, its fixture and its moment.

## Asking again without doing it again

All of that was put to use during a full dress rehearsal on a fresh test installation on November 2. A single campaign to bring up the installation’s documentation and sample apps needed a round of targeted repairs, including a documentation package built for the wrong installation. In the middle of a pass like that, a catalogue that has finished importing still leaves you asking which of the four stages you can rely on. The obvious way of settling it, running the step again, would have meant repeating the upload of 2.36 million books and a second activation on a catalogue that might be perfectly healthy.

The catalogue was already active. What we ran instead was a read-only reconciliation, which verified 2,360,648 books and 1,058,671 default filtered results without repeating the upload or the activation. The two figures do different work. The first says the published catalogue holds as many books as the fixture that was prepared, which is a statement about a count and not a comparison of every record. The second sits closer to the reader, since it goes through the default filters and the indexes behind them, and a catalogue can hold the right number of books and still have a broken default view. Neither shows that a person can search the thing and open a book, which is the browser journey’s job, and October had already shown that the journey can’t tell you the catalogue is all there. Saying a catalogue is complete and usable takes both kinds of evidence, bound to the same fixture. The same instinct, of checking what you know rather than redoing it, ran through the rest of that repair: regenerate the one wrong documentation package, deploy the three apps that were missing, bind the remaining domains without rebuilding anything, and leave whatever had succeeded alone.

A check like this is cheap to trust, since it can run after any failure and as often as you like without making anything worse. The catalogue is public to read and only the import face needs a credential, so a check like this has no reason to hold the one credential that could change anything.

The numbers describe a moment as well. I had the rehearsal installation deleted later the same day to stop paying for it, so those 2,360,648 books are gone, and what survives is the narrower and more useful fact: on that installation, on that day, two questions with known answers got those answers, with nothing uploaded to find out.