Sharing a book's text without sharing the reader's notes

Macalicious is macOS Catalina rebuilt as a desktop in the browser, developed with coding agents under my direction, and its Books app keeps two kinds of thing about a novel that could hardly be more different. The text of Moby-Dick is in the public domain, it’s the same for everyone, and it’s worth fetching once. The highlight someone puts on its first line, and the note they write beside it, belong to that person and shouldn’t be visible to anybody else. Putting those in different storage is the easy part: the book goes in a space every person shares, and the highlight goes in that person’s own. The hard part is that the highlight means nothing without the text. It’s stored as a pair of numbers, and those numbers have to keep naming the same words when its owner comes back in another browser, at another window size, possibly after the shared copy has been thrown away and fetched again from a file someone has since edited.

A place is an offset

A reader that reflows its text has no fixed pages, because what fits on a page depends on the window, so a page number can’t record where someone is. A place, a bookmark or a highlight is instead an offset into the book’s whole text, counted in UTF-16 code units because that’s how a browser counts a string. What a person keeps of a book is a small document of those offsets, one per book, so turning a page saves that document and not the whole library:

{
  "place": 1200,
  "bookmarks": [{ "at": 812, "label": "Call me Ishmael." }],
  "highlights": [{
    "id": "h-1", "start": 812, "end": 828, "color": "yellow",
    "text": "Call me Ishmael.", "note": "The first line."
  }]
}

On the other side, the server holds the text as UTF-8 parts of at most 160,000 bytes, to stay under the store’s 200 KB value limit, and the book’s index records each part’s length in UTF-16 units. A browser can therefore work out which part holds any offset before it has downloaded that part, and a part that arrives at any other length than the index announced is refused, since every offset after it would otherwise name other words.

The two sides meet only through those numbers. A request for a book’s text is answered from the shared space by code that isn’t given the person’s storage or even their session, so it doesn’t know whose browser is asking, and the reading routes work only in the space keyed by the person’s account, or by the browser’s session when nobody is signed in.

The browser journey for Books walks through the whole arrangement. A new account gets Alice’s Adventures in Wonderland from the real Book Store, turns two pages, adds a bookmark and a highlight with a note, reloads, and finds all three where it left them. A guest in a second browser gets an empty library and no reading for that book, and can still read the shared text. The same account in a third browser opens at the same words, with the same bookmark and note.

All of that rests on the text not moving, and most of the difficulty in the design comes from keeping that true while the text arrives from someone else’s server, in pieces.

Fetching in bounded rounds

Books fetches from Project Gutenberg and Standard Ebooks, and from nowhere else. For a Gutenberg book the first request reads the book’s record and refuses one whose rights aren’t public domain or that has no text, which is what makes a copy kept for everyone reasonable in the first place. Three limits of Ferrite, the cloud platform the desktop runs on, then shape the fetch. The gateway answers an API request with a 504 after eight seconds. A buffered outbound call is refused when the reply is over 1 MiB, so a book has to be read as a stream. And outbound calls pass back no response headers at all, which rules out Content-Range and ETag, the validators HTTP range requests normally lean on.

So a book arrives in rounds, driven by the browser of whoever opened it first. There’s no background job. The reader asks for the index, and for as long as the index says the book isn’t complete it asks for the part after those kept, and each such request spends at most 5.2 seconds and reads at most 2.4 MB from the source. The window shows the book once all of it is kept, so the first reader of a long book waits through every round and everyone after waits for none. A round that gets cut keeps only whole paragraphs, or whole blocks of Standard Ebooks’ XHTML, and records the byte to go on from. With illustrative numbers, a plain-text book of about 3.2 MB goes like this:

first round    GET the file, no Range
               200, stream cut at 2,400,000 bytes
               keep through the last blank line, at byte 2,398,000
               index: next = 2,398,000, and a digest of the last 4,096 bytes kept

second round   Range: bytes=2393904-4797999
               206, 836,096 bytes: 4,096 already kept, then 832,000 new
               the digest matches, and the reply ended by itself

probe          Range: bytes=3230000-3230000
               416: nothing there, so the whole reply is kept and the book is complete

Whole paragraphs matter because of what happens to the bytes next. Project Gutenberg’s text is wrapped at a fixed width, and Books unwraps prose into one line per paragraph while leaving verse and tables alone, a decision it makes by looking at the whole paragraph. Cutting inside one would put a paragraph break in the middle of a sentence and shift every later offset, so the stored text would depend on where the network happened to stop. A unit test cuts a small book at every byte a source might stop at and asserts that the kept text always equals the whole file’s.

The probe deals with the other ambiguity. Without Content-Range, a range reply that ends by itself may be the end of the file or a source sending less than it was asked for, and taking it for the end would mark a shortened book complete. So the round asks for the single byte after it, and only a 416 or an empty range counts as the end. Anything else is kept to its last paragraph, like a cut reply, and the next round asks again from there.

Inside a round the order is the familiar one. A claim key with a 15-second expiry keeps two requests from fetching one book at once, the parts go under a fresh random generation, and only then is the index replaced with compare-and-set, so an index never names a part that hasn’t been written and a slow round can’t take one backwards.

Noticing a different edition

Letting browsers drive the rounds has a consequence that matters more than the mechanics. If the first reader closes the window after round two, the index just stays where it is, and round three happens whenever somebody next opens the book, which could be much later. Project Gutenberg does correct its files, so the bytes after the resume point may by then belong to a different edition, and the usual protection, an If-Range with an ETag, isn’t available to an application that can’t see the ETag.

Books uses the bytes it already has. Each round after the first also asks for the 4,096 bytes before the place it goes on from, and compares their SHA-256 with the digest the index kept of the bytes the last round ended with. An edition with a line added or removed earlier in the file has moved those bytes, so the digest no longer matches, and the book isn’t continued. What’s kept stays as it is, the index is marked changed and truncated, and a line under the text says that it stops early because the book changed at its source. A shortened book is a real cost, but it’s the right way round, because everything already highlighted in it still names the same words.

This is a seam check and not edition validation, and it misses things. An edit further back that keeps its length, a corrected typo for instance, isn’t seen, and the text then joins the earlier edition’s words to the later one’s at a paragraph. That’s untidy, but it moves no offset, which is the property being protected. Length has a limit too: a book keeps at most 120 parts, about 19 MB of text, and one that’s longer says it has been shortened.

When the shared copy goes

While a book is kept it’s never fetched again, precisely because a fresh fetch could move offsets. The first version kept every book anyone opened for good, which bounded the space only by the size of the sources’ catalogues, some 75,000 books at Project Gutenberg alone. Two days later the cache got limits: 200 books and 150,000,000 bytes, about eight of the longest kind. Each answer notes the book’s use, at most once a minute so that reading doesn’t become a stream of writes, and when the cache goes over either limit the books used longest ago go whole, never the one being answered. The index goes first, replaced by a short-lived mark that keeps any round from publishing a new one until the parts are gone.

Eviction is where the guarantee runs out. A book that went is fetched again the next time someone opens it. From the same file the text comes out the same, because it’s a function of the bytes and not of where the rounds fell, and everyone’s offsets still work. From an edited file, offsets kept for it may name other words, and nothing moves them back. That’s where the text and label fields earn their place. They exist so the Notes list can show a highlight before the book’s text has loaded, and they also mean a note still says what its owner marked, even if the yellow now falls somewhere else on the page.

The private side stays out of all this. The cache’s record holds each book’s size and when it was last used, with nobody’s name on it, and the desktop’s shared request log writes a saved reading as PUT /api/books/reading/…, with the book’s ID cut off, because everyone can read that log. Forgetting a library raises a generation in one compare-and-set, so that every reading reads as none at once, and then overwrites the documents, so the notes are gone from the store and not just unreadable. The shared text doesn’t notice.

So the claim is a narrow one, which is what makes it useful. While a book is kept, an offset names the same words in every browser at every window size. When a round can see that the file has changed length, the book stops rather than shifts. When a kept copy has been evicted and the file has changed since, an offset may land on other words, and the saved excerpt shows what it used to mean. Searching for those words near the old offset would be the obvious way to close that last gap, and the excerpt would make it possible, but nothing does that yet.

Chris Chabot, November 2026