Real files on disk, with sync running beside them
Two systems that give tools an ordinary directory and sync alongside it: echo detection without clocks, a guarded remote commit, and measured cold-restore times.
A compiler has no interest in your replication protocol. It wants to open, stat and rename files, often thousands of them, at whatever speed the kernel can manage, and it wants a rename to be a rename. The platform around a coding agent wants something else entirely: versioned state that outlives the sandbox, can be restored somewhere else, keeps its history and accepts changes arriving from the other side. The usual way to reconcile the two is a virtual filesystem, so that every operation passes through a daemon that can record it. That design has real strengths, but it also puts the replication system in the path of every stat.
I have now built two systems that go the other way. Tools get ordinary files on an ordinary disk, and synchronization runs beside them rather than underneath them. The first was a macOS backend on a branch of a replicated storage engine, in March 2026. The second came in August on a platform I was developing: a Linux daemon that I’ll call the leaf and a small authority application, built with a mix of my own work and coding agents. They share an idea rather than a lineage: the branch never became the leaf, the leaf ports the idea onto the platform’s native revision services rather than inventing a filesystem from scratch, and the storage engine under the first one was another team’s work. They are worth putting side by side because they met the same problems and, in one important case, answered them differently.
The macOS branch replaced its FSKit and NFS backends with two loops. In one direction, a materializer took replication notifications and wrote each change to a temporary file before renaming it into place. In the other, a watcher built on the notify crate and FSEvents hashed changed content into the content store and journaled writes, deletes and renames for the trip upstream. A small SQLite database mirrored the iteration history, so version navigation, file history and rollback no longer needed a virtual filesystem to live in.
Writing real files also makes upstream metadata real. A setuid bit stored in a revision is inert until it lands on a disk, so the materializer stripped setuid, setgid and sticky bits, and it refused symlinks with absolute targets or targets that escaped the workspace. The losses ran in the other direction too. Extended attributes did not survive materialization, which meant Finder tags and quarantine flags disappeared whenever a file was rewritten from upstream.
The hard problem in any design like this is the echo. When the materializer writes a file, the watcher sees a local change and, left to itself, sends it back upstream, which notifies the materializer, and you have built a very efficient loop. The March branch kept a map shared between the two sides, from path to content hash and write time. Within two seconds of a materializer write, any event on that path was suppressed outright. After two seconds the watcher hashed the file with BLAKE3 and compared it with what had been materialized, which kept slow networks from starting a sync loop. The consequence is that for those two seconds a real edit to that path gets the same treatment as an echo.
I rejected that rule in the Linux leaf. When the leaf writes a file it records what it expects to see, which is a digest, a size and an executable bit, and the watcher compares events against that shape. A file that matches is an echo and anything else is an edit, however soon it arrives. Modification time and inode are deliberately left out of the comparison, and the code states the rule flatly: “Nothing is ever suppressed by time.” One of the watcher’s tests has a user overwrite a file right after an echo, which is exactly the case a clock would swallow.
Linux brought its own ways to lose track of things, and the inotify manual page is candid about them. Watches are not recursive, so the leaf keeps one per directory. A new directory may already contain files by the time its watch exists, so the leaf watches it first and then marks its whole subtree dirty. The kernel’s event queue can overflow, and the manual’s advice for that case is consistency checking and a rebuild. I took that advice literally: on IN_Q_OVERFLOW the leaf marks everything dirty.
The other half of the system is the authority, an application built on the platform’s own primitives: a guarded head commit, a blob store, topics and a gateway. It and the native leaf share a pure-Rust package for the codec, path rules, hashing and framing, so they interpret the same messages with the same code. The two sides trade Have and Need messages to work out which 128 KiB chunks are missing, move the chunks, send operations in batches of 48 and get back Committed or Conflict, and a reconnecting leaf is welcomed with a resume credential and the last committed sequence. The canonical decision is a single conditional commit against the encoded head, and every step after it is resumable, so a crash anywhere in that sequence leaves one question to ask, which is whether the commit happened.
The leaf keeps a durable cursor and journal so it can resume after a lost connection or acknowledgment. Its working tree is a cache. A file being present there says nothing about whether the authority has it, and that is the price of the whole design. A stat is fast because it asks nobody, so anything that needs a stronger answer has to ask explicitly, which is what the barrier messages are for. Sitting beside the filesystem sets the leaf’s other limits. It is not a filesystem, so it learns about changes only after they have happened, and it is not a uid boundary, so its check-then-use handling of ancestor symlinks won’t stop a process in the same sandbox that wants to race it.
A cache has to be refilled, which makes cold restore the measurement to look at. I measured it on August 24 by running the real leaf and authority code through a loopback gateway with injected delay, once per setting, and checking every restored file by its SHA-256. At 80 milliseconds of one-way delay, 2,000 files took 337,519 ms with one stream, 42,838 ms with eight and 11,163 ms with 32. That comes to about 169, 21 and 5.6 milliseconds per file, which sits very close to a 160-millisecond round trip divided by the number of streams. In this small-file restore, extra streams kept requests in flight while earlier ones waited for their replies. A single stream at that delay could not finish 10,000 files inside a 30-minute budget, which is why this comparison stops at 2,000.
With no injected delay, eight streams and 32 both restored 10,000 files in about 28 seconds, a floor of roughly 2.8 milliseconds per file. The authority’s per-envelope session compare-and-swap is one candidate for that floor. For tuning, the numbers pointed to a default of eight streams, with gains from 32 at the tested one-way delays of 20 and 80 milliseconds.
So the way ordinary file operations coexist with remote versioned state is that they stop sharing a path. The tools get a directory that behaves like a directory because it is one. Everything hard moves into a watcher, a journal, a handful of protocol messages and a rule that the disk is never believed on its own. That complexity doesn’t go away, but it sits where a compiler doesn’t have to pay for it on every stat.
Chris Chabot, November 2026