Taking Raft out of our configuration service

Until June 2026 our shared configuration service was also a consensus system. It kept its key/value state and its Raft log in two SQLite databases, elected its own leaders, replicated that log over our internal RPC transport, found its peers through a two-phase register-and-announce protocol on Redis, proxied REST writes from followers to the leader and shipped snapshots to S3. All of that code lived inside the application, so every client that read or changed a configuration value was relying on our implementation of leader election to be correct.

The design had a sensible origin. The service started as a subsystem inside a larger monorepo, at the point when shared configuration needed to move beyond an in-process singleton, and in March I extracted it as a service of its own. Raft is a well-understood algorithm and this was a serious implementation of it, with randomized elections, log-consistency checks, quorum commitment, snapshots, bounded proposals, joint-consensus membership changes and learner catch-up, and with adversarial and multi-process tests around them. Colleagues and coding agents worked on it alongside me.

A serious consensus implementation keeps asking for a particular kind of attention, though. The follower’s append-entries handler ran its log-consistency check even on empty heartbeats, because without it, a follower whose log had diverged could acknowledge a heartbeat and report a match index it had never verified, and a leader uses those indexes to decide what is committed. Two-voter clusters used a quorum of one, a deliberate deviation from the algorithm that sat alongside production guards requiring at least three voters. The S3 snapshots captured the key/value state without a complete log position, follower reads could be stale, and Redis discovery was not the truth about membership. Those are all problems you solve by thinking hard about replicas that disagree, and very little of that thinking is about configuration.

What the service had to own

So the useful question was which of these responsibilities the service had any business owning. A configuration service has to authenticate its callers, translate between the protocols they speak, validate what they send, encrypt what should not be stored in the clear and deliver changes to the clients watching for them. Those jobs depend on what a value means and who is asking, which no general-purpose store can know. Persistence, replication, failover, the ordering of a change stream, expiry and pub/sub depend on neither, and Valkey already provides all of them.

On June 4 I replaced the backend along that line. The commit added about 2,400 lines and removed nearly 18,000 across 52 files. The Raft log, node, snapshot and type modules went, and so did peer discovery, peer management, the Redis discovery client, S3 snapshots, the state machine and node identity; in came a state store, a Valkey client and a helper for running Valkey in development. The API servers became stateless, with no topology of their own. Existing clients kept their REST and WebSocket access, and the old cluster join and remove endpoints began answering 410 Gone, which is about the most honest thing an API can say once a concept has left the building.

Those numbers measure what the service no longer had to maintain. Somebody still had to operate Valkey, and the hard distributed-systems work now belonged to the Valkey project, which has earned the credit for it.

The other job the log was doing

Deleting the log exposed a second job it had been doing. In the old design every committed change reached one state machine in log order, which gave the service a single obvious place to keep everything derived from a write consistent. On Valkey, one configuration write touched a hash for the value and its version, a set for the namespace index, a sorted set for expiry, a counter and a stream for ordered changes, and a pub/sub channel for notifications. Issued as separate commands, those writes would let a process that died halfway notify watchers of a change that never landed or leave a version bumped with nothing behind it, and a second writer arriving in the gap could do worse.

So every mutation became a Lua script. In its August form the set script took seven keys. It read the current version, refused the write if a higher-trust source managed the value, refused it if the caller’s expected version no longer matched, stored the value with the version incremented, updated the namespace set and the expiry index, incremented the tenant’s change sequence, appended to the change stream under an ID built from that sequence, published a notification, and returned the new entry with the change. Valkey guarantees the atomic execution of such a script: while it runs “all server activities are blocked,” and its effects “either have yet to happen or had already happened.” The service still owned the rule. It ran the rule in the one place where nothing could interleave with it.

Because stream IDs came from that sequence, a watcher’s cursor was just a number, and catching up was a range read starting one past it. The stream also needed a retention policy. August’s hardening pass added approximate trimming, and with it a specific error, CHANGE_CURSOR_COMPACTED, that named the oldest sequence still available, so a client that had fallen too far behind learned it had a gap and didn’t quietly continue with one. The expiry script showed the same care on a smaller scale, deleting an entry only if its expiry was still due when the script ran, so a value rewritten a moment earlier with a later expiry survived.

Two writers and one policy

None of this protects a writer from its own stale read. An agent reads a shared policy, spends a while deciding what to do with it and writes it back, and a perfectly replicated store will faithfully preserve whichever of two such writers arrives last. In July I exposed the version check through the API: a client sent the version it read as expectedVersion, and the comparison happened inside the script. The test fires 32 concurrent writers with the same expected version at the HTTP layer and asserts exactly one 200 and 31 409s. It runs against an in-memory store that models the script’s compare-and-commit, so it pins down how the API behaves under contention, while the atomicity underneath rests on Valkey’s guarantee.

The same test file is strict about the edges. The 409 body is exactly {"error":"Version conflict","code":"VERSION_CONFLICT"}, with neither the submitted value nor the stored one in it, which matters in a service that holds credentials. A malformed expectedVersion gets a 400 before the store is touched, whether it’s a JSON number, "01", "-1" or "9007199254740992", which is 2^53, the point at which a JavaScript number stops being able to count by ones.

Compare-and-swap brings a problem of its own. A writer whose request succeeded but whose response got lost will retry with the same expected version and receive a conflict against its own write. The August hardening pass, developed with our coding agents, added idempotency records to deal with that. A pair of Lua scripts kept a digest of the request under the caller’s Idempotency-Key, marked it pending, and stored the status and body once the write completed. A repeat of a completed request got the same answer back with Idempotency-Replayed: true and no second store write. The same key with a different body got a 409 IDEMPOTENCY_CONFLICT, and a version conflict, being deterministic, was itself replayed as a 409. Records lasted 24 hours by default, clamped between 60 seconds and seven days. Recording the response was separate from applying the write. If the process died between them, a retry received IDEMPOTENCY_PENDING while the record lived; it did not recover the original outcome. The replay guarantee therefore covered completed records, with compare-and-swap still guarding the mutation.

Credentials in a shared store

Connectors kept credentials in this service, so values in sensitive namespaces were encrypted with AES-256-GCM before they left the API process, and Valkey only ever saw ciphertext for them. Which namespaces counted was configured by pattern, so this protected those values without encrypting everything in the store. The key sat in the API process’s environment, which put the protection between the credentials and someone reading or writing the store, and did nothing about someone inside the process.

My first version wrapped the result in an envelope of version, nonce and ciphertext. With nothing binding a ciphertext to its location, anyone with write access to Valkey could swap ciphertexts between keys, and they would decrypt without complaint. In August, version 2 added a key ID and bound each ciphertext to its namespace and key through GCM’s associated data, with a keyring for rotation. Old values still decrypted, and they moved to the new format the next time they were written. The binding is specifically about location: it stops a ciphertext being moved, and it would not stop an older ciphertext for the same key being put back, since the version was not among the things bound.

The same pass added provenance. A value supplied by a higher-trust source was marked as managed, and the script refused a lower-trust write against it, so an agent holding perfectly valid API credentials still couldn’t overwrite it.

The trade

Notice where all of that ended up. Encryption happened before the store was involved, and the version and provenance checks were rules the service wrote and handed to the store to run atomically. None of it needed a log or a quorum of our own, and none of it could have been left to Valkey, which has no idea what a credential is. Keeping the bytes alive through a node failure did become Valkey’s job, and that is a real trade, because the service no longer had a quorum log of its own to lean on. If you are considering the same move, read closely what your store’s replication and failover promise, because from that point on it is what your configuration service promises too.

Chris Chabot, November 2026