What “completed” means in an AI gateway
Why an AI gateway held back the terminal event until stored responses had landed, and what that release path did and did not promise about usage accounting.
By the time a streaming client receives response.completed, it has already seen the answer. The deltas have gone by, the user has read the words, and the terminal event looks like a formality. For a response the client has asked a gateway to store, it is closer to the opposite: the client may now retrieve this response by id and continue any conversation attached to it. The provider can’t make that promise. It knows its model has finished and it knows nothing about the gateway’s storage, so the completion it sends is news about the provider, and the gateway has to decide when it is entitled to say something of its own.
In August 2026 I rebuilt my AI gateway on Ferrite around OpenAI’s Responses API. The gateway adapted OpenAI and Anthropic output to that wire format and rewrote the terminal event to carry an id it had minted itself, so storage, retrieval and the client used the same identity.
For a stored response attached to a conversation, creation meant assembling the provider’s output, storing the response, its output items and the conversation turn, running the usage and spend routines, and telling the client it was done. Those records were written separately, with no transaction spanning them, so any of those steps could fail after the model had finished. If the gateway simply forwarded the provider’s terminal event as it arrived, a client could be told a response was complete, ask for it a moment later and get response_not_found, or send the next turn of a conversation that had no record of the last one.
So the gateway held the event back, with sequence accounting still advancing so that the numbers the client eventually saw stayed consistent. Then came the three durable writes. The response write was a create-if-absent with a 30-day expiry, in which finding an equivalent response counted as success and finding a different one was a state_conflict. A failure in any of the three discarded the held terminal, and the client received a failure in its place, even though the model had finished and the client had read every word. The conversation write mattered as much as the response row, because a client building an agent on this API was going to send the next turn assuming the last one was there.
Usage and spend came next. I routed terminal release through a helper that required an accounting marker, and the normal path obtained that marker only after the usage-recording and spend-settlement routines had returned. The function signature made that dependency explicit, which made accidental reordering harder. It was not an unforgeable proof: the marker’s field was visible throughout the crate, and the marker established that the routines had returned, not that every accounting write had succeeded.
Only success waited. A failure event made no promise that later reads had to honor. The same instinct shaped the rest of the lifecycle. The gateway could retry a provider only until the first output had been exposed to the client, and after that an upstream failure became stream_interrupted, because a client holding half an answer can’t be handed a second attempt as though nothing had happened. A cancel request set a cooperative flag, and finished rows were immutable from the cancel endpoint, so a completed response could not be rewritten as cancelled. In the other direction, retrieval would not show a cancellation until the runner had acted on it. In each case a status reports what has happened, rather than what someone asked for.
The storage guarantee ran in one direction only. A completed event for a stored response meant its required writes had landed, but a failure did not mean they hadn’t. With no transaction across the stores, the response could already be durable when a later step failed, and the client would be told of a failure for a response that existed. If the request was then redelivered, a nondeterministic provider could produce a different answer, the create-if-absent would find the first one, and the result was a state_conflict. A false failure costs a client a retry and perhaps an awkward conflict, while a false success sends it off to build on something that isn’t there.
Accounting had a weaker failure contract. Spend settlement retried the counter update, then tried to park an unresolved delta for a later settlement to drain. If it could do neither, it issued a warning and returned; an unavailable store could also lead straight to a warning and return. Completion could therefore follow an accounting attempt that had not retained the spend delta. Usage amounts could also be estimates, but that is a separate question from whether the record was written at all.
For stored responses, success therefore meant the response and output items, and any attached conversation turn, had passed their durable writes. Accounting ran before the announcement, but its warning-and-return paths could not support the same guarantee. A terminal event should promise only the postconditions its release path actually checks.
Chris Chabot, November 2026