A faster check has to fail the same way

One of the better performance results I can point to comes from a program that exits with an error every time. Over 24 rounds of optimization its median run dropped from 1,050.6 ms to 500.5 ms, and in all 135 rows of results recorded along the way, the exit code is 1.

That is the correct exit code. The program is a drift checker for vendored source packages, the kind of tool that runs in development and CI, and the workspace it was measured against had four problems that predated the work. Those four problems and the non-zero exit were written into the campaign’s definition of correct behavior, alongside golden hashes of the output from four commands and the tool’s test suite. An optimization that made the check faster and its report shorter would have failed, and so would one that tidied up the four problems on its way through.

I think that is most of the answer to the question of how you speed up the machinery around an agent. Checkers, redactors and event queues are the parts of a system you rely on to tell you what happened, and they sit on hot paths, which makes them tempting targets. A faster version is only worth having if it says the same things, including the unwelcome ones.

I’ll go through two campaigns and one small change, and they are not equally strong evidence. Both campaigns ran at my direction, with a coding agent working to a fixed protocol: it made the changes, took the measurements and recorded every round. Both ended on branches that were not merged, so their numbers describe candidates rather than anything people on the main line received.

Where the time went

The first campaign targeted the checker in my Ferrite project, which compares what 28 source packages should have put into 41 consumers, across 117 mounts and about 1,300 vendored Rust files, with what is actually there. For a tool that hashes everything it reads, hashing is the natural suspect. The profile gave it 0.5 percent of self time, and JSON parsing 1 to 2 percent. Directory reads took 64.5 percent. On that machine, a two-vCPU container on a FUSE overlay filesystem, the work was bound by filesystem metadata rather than computation, and the largest single hotspot was a walk looking for misplaced manifests: 1,680 readdir calls over 11,046 entries, for about 265 ms. There wasn’t even one enormous directory to skip, because the cost came from breadth, with 945 of those calls spread across the apps. The next hotspot was plain repetition, each package file being read again for every consumer that mounted it.

The rules for changing any of this were strict. Each round changed one thing and measured it in interleaved A/B pairs. At a coefficient of variation of 1.1 percent, a change had to move the median by about 3 percent, some 30 ms, to be distinguishable from noise at all, and the campaign demanded 5 percent, in every pair. Of the 24 rounds, 10 were accepted, 8 rejected and 6 changed nothing. The first rejection is a good one: recursive readdir, the obvious tool for a program that spends its life walking directories, came in 2.2 to 3.6 times slower. Three other candidates would have saved 4.2, 3.2 and 2.0 percent, and stayed out because they were under the gate.

What survived was unglamorous. Caching package bytes took the run from 1,220 reads to 285 and saved 11.6 percent in its round. Concurrent directory reads saved 11.2 percent. Other rounds read only the first 4 KiB of a file where the head was all that mattered, issued reads in batches of 16 and stopped recomputing digests for content already in memory. The final comparison, 40 runs against the original 20 on the same host with warm caches, gave 500.5 ms against 1,050.6 ms, a reduction of 52.4 percent. That number belongs to that machine, because a workload dominated by readdir on a FUSE overlay may move quite differently on a local disk.

What the gate can’t see

The gate can tell you that a change is faster. It can’t tell you the change is the same, and that is where the campaign’s best details are. The 4 KiB read drew a review finding: a single read call may return fewer bytes than you asked for, so code that treats one call as filling the buffer can miss a marker sitting right there in the file. The fix was checked with a scan of 1,912 files and a simulated 7-byte short read. A later memoization came with a differential check, and the version without its guard failed 200 times out of 200. Those 200 failures say more than any pass count could, because a check you have never seen fail has not told you much.

Concurrency raises a subtler question. When a tool dies on the first fatal problem it meets, the order in which it reaches things decides which problem you hear about, which makes that order part of its observable behavior. One round produced no code at all, only an analysis of the order in which the tool’s fatal errors fire.

The most useful of the no-change rounds came near the end. By then the walk cost about 160 ms against an estimated floor of about 121 ms, at 72 microseconds for each readdir. One proposal was to skip 427 web directories. The campaign treated that as a policy decision for the project’s owner, which is to say me, and left the code alone. That is exactly where it belongs. A misplaced manifest in a directory the walk no longer visits is a problem the check no longer finds, and a change that does this is a decision about what the check is for, whatever it does to the timings.

When the behavior is a security property

The second campaign, at the end of August 2026, worked on the runtime of a coding agent, where some of the behavior to preserve protects secrets. Terminal output passes through a redactor, and the existing code did work that grew quadratically as output accumulated. Making it incremental cut its benchmarks by 29 to 31 percent. But incremental redaction is the kind of change that can get faster by getting worse, since a redactor that only looks at the newest chunk is quick and misses any secret that arrives in two pieces. What makes the change credible is the differential check behind that number: 7,875 sequences, 72,480 chunks and 1,275 boundary cases, old code against new, with no mismatches.

The rest of that campaign shows the two ways an optimization can mislead you: the gain can be noise, or the gain can be real and the behavior different. A cache that looked useful fell into the first group, with one of its A/B pairs coming out inverted, so it was rejected and its diff kept on file. Review found the second kind, in two problems no benchmark would have caught: a block object shared between envelopes, and a target that was read-only only in TypeScript’s type system, which means nothing at run time. A good deal of micro-optimization comes down to sharing or reusing something, and sharing changes behavior as soon as anything writes to it. The first became a factory producing one per envelope, the second got Object.freeze, and both were measured again and stayed within 3 percent.

These were microbenchmarks on a two-core VM, at 60 to 600 runs, with no attempt to multiply percentages across scenarios. They show that those paths got cheaper. They do not show that an agent finishes sooner, costs less or does better work, and I wouldn’t claim that they do.

An argument is not a measurement

The third change is different in kind. In May 2026 I changed the event queues in a TypeScript streaming client to stop calling shift() on an array. Repeatedly removing the first element makes the drain depend on how the engine handles the remaining backlog, with the risk of doing that work again for every event. The new version keeps a head index, clears each consumed slot so the event can be collected, and compacts the array once the head has passed both a threshold, 1,024 in one client and 64 in the other, and about half the array’s length. The realtime client’s waiter lists got the same treatment.

I believe this is an improvement, but what I have is an algorithmic argument and nothing more. There is no benchmark behind it, and engines have their own tricks for shift(). It does not bound the queue either: a consumer that falls behind still accumulates events and nothing pushes back. And it too sat off the main line.

It would be easy to put 52.4 percent, 29 to 31 percent and the queue change into one sentence about a faster platform. They are three different kinds of claim: a same-host median against a golden oracle, a microbenchmark range with a differential check, and an argument from a data structure. Collapsing them would do to the reader what the oracle stopped the optimizer doing to the checker, which is to improve the report by changing what it says.

Chris Chabot, November 2026