Twenty-one workspaces sharing one filesystem flush

In the summer of 2026 the agent platform I was working on gave every coding workspace its own storage stack: a FUSE daemon serving the agent’s working directory, a content-addressed store and a segment log behind that daemon, and a Litestream sidecar replicating the agent’s SQLite state. Twenty-one workspaces meant twenty-one independent daemons. What they shared was the filesystem, because all of those instance directories lived on one XFS filesystem inside a container overlay, on a host with 8 vCPUs, 30 GB of memory and a single NVMe drive. When I ran them side by side in early August, the median time for a fixed workload went from 11.5 seconds with one workspace to 63.9 seconds with twenty-one, and all 93 sessions in the campaign finished without an error. Nothing was failing. Everything had just become more than five times slower.

A slowdown like that diagnoses nothing by itself, since twenty-one of anything on one drive ought to be slower than one. The useful question was how much of that 5.6× belonged to the hardware and how much belonged to us, so I built a stress harness to find out. A stub stood in for the agent, which kept model inference out of the picture, and each workspace wrote 8,000 files totaling about 600 MB, most of them under 4 KB and a few as large as 8 MB. It then read everything back, overwrote a fifth of the files, renamed a tenth, deleted 30 percent and walked what was left. The seeds were deterministic, so runs could be compared like for like.

The decision that turned this from a complaint into a diagnosis was to run the workload three ways: through the workspace filesystem, on the native filesystem with ordinary buffered writes, and on the native filesystem with an fsync for every file. The third arm was the one that mattered. Buffered writes make a flattering comparison because they promise nothing about durability, whereas the fsync arm was matched to the durability the workspace filesystem offered, and it degraded from 31.1 seconds to 42.3, a factor of 1.36. So the drive could absorb twenty-one workspaces making durable writes. The workspace filesystem started 2.7 times faster than that control and finished 1.5 times slower, which put the problem in our stack and showed that it grew with the number of neighbors.

The reason it started faster was also the reason it finished slower. Storing a file in the content store was a create-temp-and-rename with no per-file fsync, and durability was deferred to a group commit that closed at 512 records or after 50 milliseconds. Just before flushing a group, the daemon called syncfs(2). The engine’s owners had added this in June as the fix for a small-file storm, with a design comment that summed up the idea as “One syncfs, not N per-file fsyncs.” In isolation that is a sound trade, as the solo numbers show, but it assumed that every daemon had a filesystem to itself. The manual page is clear about what happens when that stops being true: syncfs synchronizes the filesystem containing the file you hand it, and on Linux it provides the same guarantees as calling fsync on every file in that filesystem. With twenty-one daemons on one filesystem, each of them could ask the kernel to flush all twenty-one daemons’ dirty pages as often as every 50 milliseconds.

The measurements around that mechanism fit it closely. Dirty memory in the workspace arm peaked at about 25 MB, against 4.3 GB for buffered native, so the page cache was being emptied nearly as fast as it filled and had no chance to absorb anything. The drive was not out of capacity, with a 95th-percentile write wait under 3 milliseconds and utilization at 74 percent. The load average sat at 20.8 while only about 2.7 of the 8 cores were busy, which is what a machine looks like when almost everything is waiting and nothing is saturated. The last piece was inside the daemon, where FUSE dispatch defaulted to a single thread, n_threads.unwrap_or(1), so every operation in a workspace queued behind whatever flush that thread was sitting in. Comparing matched seeds alone and under load, median write time went from 0.29 to 0.89 milliseconds and the 95th percentile from about 1 to about 12. Median read time went from 0.06 to 0.93 milliseconds, fifteen times slower, for an operation that has nothing to flush.

N daemons each flushing the pages of N daemons sounds quadratic. What the numbers showed was a ceiling: aggregate write throughput flattened out at about 400 MB/s, at nine workspaces and again at twenty-one, while each workspace’s share fell from 135.7 MB/s alone to 19.1.

I could change where the engine kept its data. Since syncfs takes its scope from the filesystem, changing the layout changes what a flush means. I moved instance state and a shared content pool onto two dedicated loop-backed ext4 filesystems. That is two filesystems in total and not one per workspace, so the daemons still flushed one another’s pages and the growth was bounded rather than removed, but a flush now covered the storage it was meant for instead of everything on the host’s one filesystem.

The first version of this looked excellent until it was asked to do the thing it was for. On ordinary buffered loop devices the median was 10.5, 13.1, 16.1 and 18.6 seconds at one, three, six and nine workspaces. At twenty-one it regressed to 69.9 seconds, worse than where I had started, with the loop device pinned at 100 percent utilization at 243 MB/s. Switching the loop devices to direct I/O gave a little back at the low end, 22.5 seconds at nine, and in its one cold-pool run at twenty-one brought the median down to 38.7 seconds, with the slowest at 42.3 and no errors across the 113 validation sessions. That is about 39 percent off the median for this stress workload on this host, and it is the layout I deployed and checked through the real stack as well as the harness.

On the dedicated layout, native work took 23.1 seconds at twenty-one workspaces, against the contained system’s 38.7 seconds. That absolute comparison puts the contained system at 1.67 times the native floor.

Containment was never the real fix, because the assumption behind the flush was still wrong and the single dispatch thread was still there. My proposal for the engine led with replacing syncfs with fdatasync or sync_file_range over each batch’s own files and making dispatch multi-threaded. The engine’s owners landed their own fix the next day, replacing the filesystem-wide flush with a per-entry drain. The 39 percent belongs to the layout change, which came first.

Narrowing the flush also exposed a durability dependency in the shared content pool. One process can deduplicate against an object that another process has published but not yet made durable, and a per-entry drain has to track that case explicitly. The old whole-filesystem syncfs covered that cross-process window implicitly. An opt-in experiment of mine on a side branch shows how much the narrower version has to carry. To confine a commit’s synchronization to its own content objects, it tracked deduplicated objects as well as newly published ones, ran fdatasync on the files and fsync on the publication directories before the metadata commit, and restored the pending paths if a sync failed.

What I would carry to other systems isn’t the 39 percent, which belongs to this workload on this host. When a durability primitive takes its scope from the filesystem, the deployment layout is part of its design, and an optimization that was right on the day it was written can stop being right without a line of it changing.

Chris Chabot, November 2026