# Making a search index prove itself at query time

Author: Chris Chabot
Canonical URL: https://chabot.dev/blog/making-a-search-index-prove-itself-at-query-time/
Published: 2026-11-03

A search engine that treats every posting as a lead, publishes updates by stage-then-flip, and checks each candidate against one served version before its ACL runs.

A search index is a collection of statements about documents, each of them true at the moment it was written: this term appears in this record, at this version. Records keep changing after that. An indexer can die with half of a document's new postings written, and a permissions change can leave a perfectly well-formed posting pointing at a document its reader should no longer see. A query that takes the index at its word can then return a result assembled from two versions of the same document, ranked with statistics that describe neither.

I led the integration of this search engine in August 2026, working with coding agents on an imported implementation and the platform's guarded-commit and storage primitives. It ran behind a node service and a command-line tool. Every index entry is a candidate that has to be confirmed against the current record when a query runs. With one indexer writing each engine, updates follow a stage-then-flip protocol, so that a crash at any seam serves either the old version or the new one, fully, and never a mix of the two.

## Decisions in one place, entries in another

The split I settled on uses two of the planes Ferrite, the platform I own, offers for keeping state. A decision plane with guarded commits holds whatever has to be true: document records, engine metadata, corpus statistics, cursors, rejection records and the indexing worklist. The store holds the index entries: postings, filters and document bodies. An early or out-of-date posting can remain in the store without deciding what the query serves, because the document record determines which version is eligible.

Each document's record says whether it is visible, which version is indexed and what is staged to replace it, and it holds a footprint of the served version: version number, epoch, terms, filter entries, field values, length, title, ACL, source, update time and body digest. Everything a query needs to judge a candidate therefore sits in one place and belongs to one version.

## Four steps to publish a version

An update goes through four steps, any of which might be the last thing the process ever does.

Admission validates the document and stops early if its digest shows nothing has changed. It computes the index entries before writing anything, so a document too large for the index is turned away without leaving a body behind. The body then goes under a versioned key, leaving the old version's body where it was, and one guarded commit records the staged footprint and adds the document to the pending worklist. If the worklist is full the caller gets `BACKLOG_FULL`, which is back-pressure rather than data loss.

Staging works only from what admission made durable. The indexer checks the stored body against the digest in the staged footprint, returning `REBUILD_REQUIRED` if they disagree, recomputes the entries from the body and the footprint's title, and writes them in batches. The new postings are now physically in the index, where any query can read them, and they don't count yet.

The flip is the commit point. One guarded commit makes the staged footprint the indexed one, pushes the old footprint onto the record's garbage-collection list and updates the corpus statistics, so the numbers [BM25](https://en.wikipedia.org/wiki/Okapi_BM25) scores with move at the same moment as the served version.

Collection then deletes the old entries, key by key because the store has no batched delete. Until it finishes, the old postings remain in the index and are ignored.

Suppose an update replaces `oldterm` with `newterm` in a document. This is what a query sees if the process stops after each step:

| Stops after | Durable by then | Query for `oldterm` | Query for `newterm` |
| --- | --- | --- | --- |
| Admission | New body, staged footprint, worklist entry | Old version | No match |
| Staging | New postings as well | Old version | No match |
| Flip | New footprint served, old one queued for collection | No match | New version |
| Collection | Old postings gone | No match | New version |

The engine's crash test stops the protocol at the first three of those seams, hands the worklist to a fresh indexer and asserts exactly this: the old version fully served until the flip, the new one fully served after it, resumption idempotent and statistics in agreement. The fresh indexer needs nothing from the process that died, because the worklist and the staged footprint say what is left to do.

## What a query checks

A query starts with candidate documents from postings, and at that point it has only leads. Each candidate then goes through the record in a fixed order: the document has to be visible, it has to have an indexed version at all, which excludes anything admitted but never flipped, and the footprint has to be the one for that version. Negated terms are checked next, ahead of any access decision, and each posting then has to carry the indexed version and name a term the footprint contains. Only after all of that does the ACL run, against the served footprint, followed by phrases, filters, scoring and facets.

Run the stale cases through that. A posting for `oldterm` left behind after the flip has the wrong version and names a term the footprint no longer contains. A posting for `newterm`, staged ahead of the flip, fails the same two checks from the other side. A deleted document fails at the first check whatever its postings say, because a delete writes a tombstone on the record and takes effect immediately. And since the ACL comes from the served footprint, a posting has no way to carry an old permission forward.

That last property has a consequence worth knowing about. A permissions change that arrives with a new version of a document takes effect at the flip, along with the rest of that version, because "old or new, never a mix" covers the ACL as well as the text. If something has to disappear right now, the tombstone is the tool.

The same instinct shows up in smaller decisions. Facets are counted only over documents the principal can fully access, since a count is also a disclosure. A term longer than 64 bytes is skipped and never truncated, because truncation would alias distinct terms and make the index wrong on purpose. Continuation cursors are [HMAC](https://datatracker.ietf.org/doc/html/rfc2104)-signed and bound to the engine, the query, the principal and the epoch, so a cursor cannot be carried to another user or to a rebuilt index. A query that sees the epoch or sequence move while it is running retakes its snapshot, up to three times.

## Refusals with names

Confirming candidates against records costs reads, and the engine doesn't pretend otherwise. Every accepted query is index-backed and budgeted, and anything else gets a typed error from a closed vocabulary with stable codes, such as `QUERY_TOO_BROAD`, `CURSOR_EXPIRED`, `REBUILD_REQUIRED`, `CONFLICT` and `BACKLOG_FULL`. A stable code gives the caller something to act on, where a quietly plausible partial answer would give it nothing to debug.

`CONFLICT` guards a rule the whole design leans on. One indexer writes per engine, which is the platform's standing discipline for read-modify-write keys like the worklist and the document records, and it would be easy to trust that rule and stop there. Every decision commit is guarded on the record's raw bytes regardless, so a misconfigured second writer degrades into failed guards and retries and, after 32 attempts, a loud `CONFLICT`, where it could otherwise have corrupted records in silence.

Once an index is something you check against records and can rebuild from them, you can change how retrieval works while keeping the record's served version and permissions authoritative.