Why opening a conversation should not wait for its whole history
A page asking for twelve entries could still wait 31 seconds. Pagination limited the answer and not the work behind it, so the design serves the first view from read models the writers maintain.
At the start of November, opening a project, a thread or a worker in the web interface of our agent platform took between 15 and 95 seconds, and some opens never finished at all. On 2 November I set a goal of a few hundred milliseconds at most. We wrote that down as a target that could be checked: the gateway answers the first paint of each of those three views in under 300 ms at p95, and the page is usable at that point, with its header, counts, newest items and composer. Everything else arrives afterwards, in pages or as pushed frames.
Two explanations that would have been convenient were already used up. The gateway’s local SQLite store had grown to about 50 GB and had been freezing the event loop, but it was swapped out that same morning, and by midday the file was down to 102 MB. Those stalls explain the freezes of 1 November. They do not explain the slow opens we were measuring on the 2nd. We had also already paginated. In October, long conversations started loading from the newest page, with older messages fetched by cursor, durable position indexes selecting rows before their bodies were read and routes rendering recent content before slower inventories finished. That work was worth doing, and by November the page really was asking for only twelve entries. One such read took 31 seconds. Pagination limits how much comes back, and it says nothing about what the server has to finish before it can send anything.
What twelve entries were waiting for
Finding out took nothing more than reading what the development host already recorded: about twelve hours of per-method timing lines, the event-loop stall records, the per-project commit logs, and two live reads made the same way the page makes them. The timing lines could not tell a web call from an agent or administrative call, and agents use some of these methods too, so their figures describe what each read path cost rather than what any one person waited for. What they showed was time going to work that had nothing to do with the view, in four different ways.
Reads of a project queued behind its writers. A read that found the project’s store not current ran on the project’s write chain, behind reconcile commits that took 20 to 150 seconds against a 20-second budget, and up to 1,557 seconds on one very large project. The reducer inside those commits also awaited record reads while it held the chain, and in one pass spent 27.6 seconds on 22 inspection reads and 27.7 seconds on a single page rotation. That twelve-entry read finished within seconds of the end of its project’s 59.5-second reconcile pass, which is what a read waiting on the write chain looks like, and the same project had logged six passes of between 20 and 152 seconds in the half hour before it.
Project reads also rebuilt the whole project and then threw the result away. A read that accepted stale state never installed what it had loaded into the held cache, so every open of a project that was not already held repeated the cold fill: the control record, then every item at rest with its evidence, review cycles and context. The code records 59 to 127 seconds for one such fill on that same large project, and the project snapshot call reached 391 seconds that morning.
Opening a thread warmed its entire transcript from S3 before showing twelve items. The first page of history and the snapshot for a stream subscription both waited for a full-thread warm, one segment at a time, up to a 25-second deadline. Separately, every call that named a thread first waited up to 2.5 seconds for a warm of its turn records, including the read of the thread’s own metadata and the reads of its background tasks and jobs, none of which read a single turn row. A thread open sends three of those.
The fourth was the old problem on its way out. Event-loop stalls of 15 to 71 seconds, 28 of them across 1 and 2 November, delay every request in flight. Most came from the SQLite store and went with it, and the last came from commit finalization and parsing a pack from S3.
The logs made these causes easy to tell apart, once we looked at where the latencies sat as well as how high they were. Of the 95 slow calls of those two thread methods, most ended at the 25-second cap, with history reads bunched between 23 and 25.6 seconds and subscriptions at about 25.1. Background task reads had a window p99 of about 2.5 seconds. A latency distribution that sits on a constant is showing you a deadline rather than a workload. Call counts pointed the wrong way entirely: the busiest methods, at about 18,500 calls each, were a poller asking after deleted threads and getting a quick typed not-found, which had nothing to do with navigation.
Not everything was about time. A snapshot of a project already held in memory came back in at most 31 ms and weighed 259 KB for one work row, 120 KB of it being all 84 of the project’s decisions, with nothing bounding them. The answer was quick and still far heavier than a header, some counts and the newest items need to be.
Maintained by the writers
Those mechanisms have a common shape: every first-paint read was built on demand from the authoritative stores, and so it inherited whatever those stores were doing. The design replaces that with read models maintained by the writers, a project header for the project view and a transcript tail for the thread and worker views. The commit and event paths already see every change, so they keep these current, encode them once and serve them from memory, and persist them in batches to the shared-record store so that they survive a restart. They are never rebuilt on the request path. Reads stop taking the write chain, the client paints from its own cache and revalidates, and every first-paint method gets a hard server deadline after which it returns what it has. Older history still comes by cursor, and later changes arrive as pushed frames.
This separates authority from presentation, which the old path had welded together. The stores still decide what has been committed and what the history contains. The header and tail only have to be good enough for a person to see the work and start typing, and they are allowed to be a moment behind while the complete state catches up. The writers take on more responsibility in exchange, which is where the performance rules I had set for the whole system two days earlier come in: write rarely and in batches, encode an event once and share the bytes, and keep long work off the event loop.
The design’s first step is not a read model at all. A target stated in terms of people opening pages cannot be judged from timing lines that mix agents in and do not name a project or thread, and two of the project read methods turned out to have no web caller at all. So the telemetry gets fixed first, and each step after it is small enough to land and release on its own.
Here 300 ms is still a target and not a result, and what the diagnosis establishes is where the problem sits. Every message on that first page passed through the system when it was written. The writer had the header and the tail in hand at that moment and let them go, which left each reader to rebuild them from the authority and to wait behind the writer while doing so.
Chris Chabot, November 2026