Testing the promises an API actually makes
How to test what a multi-service JSON-RPC API promises: coverage from its advertised methods, hostile payloads, concurrent writes and literal replay checks.
Most of what a multi-service API promises doesn’t belong to any one service. When two clients write with the same expected version and exactly one of them is supposed to win, that outcome depends on the handler that checks the version, the store that applies the write and the transport that carries the conflict back to the loser. When a client replays a burst of events with a limit, the order, the limit and the absence of duplicates only hold if everything between the stream and the socket agrees on them. Each of those pieces can have a respectable unit test suite and a green health endpoint while the promise itself is broken, because the unit tests check the pieces and the health endpoint checks that a process can answer a request.
The platform I work on is an agent system whose JSON-RPC API sits in front of conversations, event streams, stored resources and workspaces. In the spring of 2026 I built two batteries of tests aimed at that boundary. The first drives the API as a whole and tries to make it break its contracts with malformed input, concurrent writes, stream bursts and hostile content. It exercises the API’s own layers through a local server, with in-process stubs standing in for downstream services and automatic execution switched off. The second goes service by service through the native handlers, parsers and transports. Both are organised around one question: given what this thing says it does, does it do it?
The first decision was to take the scope of the test from the server instead of from me. The server’s initialization response lists what it supports, and the runner’s coverage check is a set difference against that list, so a method that gets advertised without being exercised shows up as a difference. Test suites tend to decay in the opposite direction: the API grows, nobody remembers to add a case, and the suite stays green while covering less and less of the product. Before the sweep, the runner seeds every surface, since a call that returns “not found” from an empty store has told you nothing about how the method handles a bad payload. Then it calls the methods and holds the whole run to two conditions, no RPC errors and no call slower than five seconds. That is a modest oracle, but it applies everywhere. These payloads are hostile but valid, so the methods have to accept them without falling over or sitting there chewing on them.
The hostile content is a fixed list of eleven strings: the empty string, a single space, whitespace with tabs and newlines, 16,384 copies of the letter a, control bytes including a NUL, a combining accent, a right-to-left override in front of secret.txt, a format-string probe, a script tag, the obligatory '; DROP TABLE threads; -- and ../../../etc/passwd. They are joined into a single payload with a case number. This is a corpus and not a fuzzer, so it finds the kinds of trouble somebody thought to write down, and in exchange a failure is the same failure every time you run it. Where a payload travels through a resource or a workspace, the check is round-trip preservation: what you read back has to be what you wrote. The script tag comes back as a script tag. A system that quietly cleans stored content has corrupted it, however good its intentions were.
The same bytes, different verdicts
Testing at each boundary matters most here, because whether a string is hostile depends entirely on where it is going. One native server has to accept paths with spaces, emoji, deep nesting, semi;quote' and a directory literally named literal..dots/, while refusing ../secret.txt, ..\secret.txt and an embedded NUL with an error that names the escape or the NUL. The stream handlers refuse ../escape as an invalid target. The Telegram bridge has to keep an account called ../unsafe account from turning into an unsafe state path, and yet a prompt that merely looks like a traversal has to reach the agent verbatim, because in a prompt those characters are just somebody’s text. No single sanitizer can satisfy that set of cases. Strip every .. and you break a legal directory name and rewrite what a user said; strip nothing and the path cases escape. The behaviour has to be right at each boundary separately, so each boundary has to be asked separately.
Transport errors get the same literal treatment. The JSON-RPC 2.0 specification reserves -32700 for a parse error and -32601 for an unknown method, so the test sends unparseable input and expects the first, calls a method that doesn’t exist and expects the second, sends a GET and expects a 404, and then it pings. The ping carries the most weight, because a server that returns all the right codes on its way down has not kept its contract. Further along, the native battery requires malformed streaming frames to parse to null, the voice parser to get tables of valid and invalid frames right, and the SDK to ignore a duplicate sequence number, advance its checkpoint and send in exactly the expected order.
The concurrency checks are useful for the same reason: each is a fact a client could verify without knowing anything about the implementation. Twenty-four groups of operations run at once, each creating a thread and writing a distinct resource and a file in a shared workspace. Afterwards the thread IDs have to be unique, each resource and file has to retain its own contents, and a listing of the shared resource prefix has to equal exactly what the groups wrote, with nothing extra and nothing missing. Two contenders write with the same expected version, one succeeds and the other gets “Version conflict”. Replaying 250 events with a limit of 80 has to return no more than 80, sorted, with unique IDs. With one concurrent turn allowed per project, a second turn is refused and cancelling the first releases the slot. A different user asking about that thread gets “Thread not found”, and a different user trying to drop someone’s stream subscription gets “Subscription not found”, so neither answer confirms that the object exists. The limiter’s calculation gets a deterministic check: at three requests per second, it allows the first three of six checks and refuses the rest. The refusal must name the global identity scope and carry a positive retry-after, and a separate assertion pins the dedicated error code.
The most compact of these fits on one line. Publish sequence numbers 5, 1, 3 and 2 into a history capped at three entries, replay after 2, and the answer must be exactly [3]. The 5 arrived first, so the cap evicted it, and 3 is the only thing left past the cursor. Evict by the wrong key or treat the cursor as an offset and the array comes out different. An oracle that literal needs no knowledge of how any of it is built, which is what lets it survive a rewrite of the thing underneath.
Chris Chabot, November 2026