An interrupted model call still has to be accounted for

A client loses its connection partway through a streamed model response. It has some of the words and no terminal event, and from where it sits it can’t tell whether the model carried on to the end, whether the provider will bill for the work or whether anything in between recorded what happened. A gateway in the middle doesn’t get to share that uncertainty for long. The same request may well arrive again in a moment, and sooner or later someone will read a figure off a statement that is supposed to say what this call cost.

The two obvious rules are both wrong. Charge only for responses the caller received in full, and you absorb the provider’s cost on every dropped connection and hand a free answer to anyone who disconnects just before the end. Charge for everything you dispatched, and a provider that refused the request before doing any work still costs the customer money. Neither rule looks at the one thing that decides the question, which is what the provider may have done, and that reaches the gateway by a different route from anything the caller saw.

In August 2026 I rebuilt an AI gateway as an application on Ferrite, and I’ve written separately about what “completed” means in a gateway like that. Accounting was the weak part of that version: settlement retried a counter update, tried to park an unresolved delta for later and, if it could do neither, issued a warning and returned. That autumn, working with agents, I ported the gateway onto Ferrite’s core as a resident service, a long-running process beside the fleet with its own identity, rather than a constrained application. It serves OpenAI’s Responses API as a buffered reply, over SSE and over WebSocket, reaches providers only through the platform gateway’s egress path and charges each request to a Ferrite account through the platform’s own statement ledger. That ledger belongs to the core, which I build on rather than own. What I’m describing here is the service in front of it, which decides what to ask the ledger for and when, and which I’ll call the front.

What the caller’s connection tells you

Every request becomes one operation with a durable record, filed under its account and response id. Before dispatch the front writes the estimate to the account’s statement, so the credit is spoken for before a provider hears about the request. Requests that bring their own provider key get the same record and touch no Ferrite credit.

From there the caller’s connection matters very little. A closed channel to the client kills one thing, the tap that publishes events to that client. The fold of the provider’s stream and the buffered result keep running, so a caller that hangs up doesn’t stop the response from being assembled or the provider’s final usage from being read, and the call settles at its real catalog price.

It took a review to get that boundary right. In an earlier revision, whether anything had reached the client decided both whether a failed attempt could be retried and whether it was refunded, which is the obvious rule and the one the August gateway used for retries. An independent review found what was wrong with it. A buffered call exposes nothing until it finishes, so a provider that accepted such a call and then failed looked like a call that had cost nothing. It would have been refunded and could have been sent again, even though the provider might have been generating tokens and billing the platform for them. A timeout while waiting for the response head had the same problem. Exposure isn’t the boundary for money or for replay, so the front now tracks a separate fact, whether the provider may have performed paid work, and sets it as soon as a 2xx head arrives or the wait for one ends ambiguously.

One decision for a failed call

When a call fails, one small function decides what happens to the reservation, and its test walks the complete truth table. If usage is known, because the terminal event carrying it has already arrived, the call settles at that usage whatever else happened. Otherwise, if anything was exposed or the provider may have worked, the front retains the reservation and marks the usage unknown. Only if nothing was exposed and the provider cannot have worked is the reservation refunded whole. Retries follow the same facts: a failed attempt may be sent again only if the error is retryable, nothing was exposed and no paid work can exist.

The middle outcome is the one you have to be willing to defend. The estimate is deliberately pessimistic, pricing text at two bytes a token and assuming the whole output allowance, so a customer whose stream the provider cut can pay more than the call would have cost. I still prefer it to the alternatives. Repricing from partial input or inventing an output count would produce a figure that looks measured and isn’t, and a refund would have the platform absorb whatever the provider billed. The rule is never more than the reservation and never zero, and the record says usage_unknown, so nobody mistakes the figure for a measurement.

“Cannot have worked” needs a precise definition, and the front’s is strict. An error counts as pre-send only if it is positively recognizable as one: a connect or TLS handshake timeout, a refused connection, an unreachable network, a name that didn’t resolve, a destination the gateway wouldn’t dial. A timeout waiting for the response head, a closed or malformed head, a cut or silent body and any error text the code doesn’t recognize all count as possibly paid. A provider’s own non-2xx answer is handled separately, since that is the provider refusing the work, and a 429 or a 5xx is retried within bounds and then refunded. So a model that takes more than 10 seconds to produce a response head, or goes quiet for 30 seconds in the middle of a stream, fails as a possibly executed call: it isn’t retried, and the estimate is retained.

That judgment also has to survive a crash, so dispatch is recorded before anything is sent, and an operation that can’t record its dispatch isn’t dispatched. The background sweeper, which runs at startup and every 60 seconds, can then tell the two cases apart for a process that died. A reservation past its deadline with no dispatch marker was never sent, so it is refunded, and one with a marker retains the estimate.

Fixing the intent first

A process can die anywhere in that sequence, so something has to be able to pick up the work: a retry of the request, the sweeper or the same process after a restart. If each of these worked out for itself what the operation deserved, two of them could reach different answers, because evidence can turn up between one attempt and the next. One process refunds, its successor sees the usage arrive and settles, and the customer’s statement now says both. The ledger’s guarantee doesn’t prevent that. It writes each statement line exactly once, and a settlement’s remainder credit and a refund’s credit are two different lines with nothing fencing one from the other.

So the front writes down its decision before it asks. A compare-and-swap puts a settle intent or a refund intent on the record, only onto a record carrying neither, and an intent is never rewritten. Whoever picks up the operation afterwards decides nothing: it replays exactly that call and finds its own line Present. Usage gets the same treatment, persisted to the record the moment the terminal event arrives, before a cancellation, a provider disconnect or a crash can select how the operation settles. A refund intent also refuses dispatch, so a refunded operation can never be sent afterwards.

Confirming a line has a failure of its own, and it shows how much care an honest unknown takes. The ledger keeps the most recent 200 statement lines per account, shared with every other charge, and a busy inference account goes through that in about a hundred requests. Each step records the statement’s next sequence number as a floor, in the same write that fixes the step. While the ring still retains everything from the floor on, a missing line is provably missing. Once it doesn’t, the ledger answers that it can’t tell and writes nothing. A less careful system would read that as “never written” and charge again, or as “written” and move on. The front marks the operation indeterminate, leaves it open, reports it on every sweep and asks the ledger nothing further.

An operator can then list those operations, read the evidence for each and record a decision, either settle_as_charged, which accepts the statement as it stands and writes no line, or refund, which credits the account once, with a note and the response id repeated as confirmation. The decision is written to an audit record before anything it implies is done, and it is never rewritten. A person deciding a genuine unknown on the record is a better outcome than a rule that settles it silently in someone’s favor.

The caller’s side of a retry

The caller has its own way through the uncertainty, which is to send the same request again with the same Idempotency-Key. The key is bound to a digest of the canonical request: the account, the route and the body, with member order normalized and any credential carried in the body removed, so the model, the input, every parameter and the stream flag are all part of it. If the digest matches, the front replays the recorded answer with no provider call and no ledger effect. A streamed request replays as an event stream from the frames it recorded, up to 2,000 frames or 512 KiB, beyond which only the terminal event comes back. An identical request that arrives while the first is still running waits up to 20 seconds for its outcome and then gets 409 idempotency_busy. In each case the caller hasn’t had to know how far its first attempt got, which is exactly what it couldn’t find out.

The strict parts of that contract are the ones protecting the money. The same key with a different digest gets 409 idempotency_key_reused before anything is dispatched or charged, and that includes a caller that lost a stream and tries again with streaming switched off, since the stream flag is in the digest. A key that’s sent but can’t be used, because it’s too long, contains a control character or arrives as an empty or repeated header, or on a WebSocket as a number, null or an empty string, gets bad_idempotency_key and is never treated as absent. That can look pedantic until you follow the alternative. The request would run with no key, the caller would believe it had protection it didn’t have, and the retry after a dropped connection would be a second charge. Only a request that sends no key at all runs without one.

Reading the number back

None of this means much to a customer until it arrives at a number they can check. The failure paths are covered by the service’s deterministic suite, with in-process fixture providers and billing failure races, since a shared hosted service isn’t deliberately broken to repeat that proof, but a fixture isn’t a provider. The number itself was checked in a dress rehearsal on a fresh installation, where four real provider calls and eight authentication and isolation checks passed and the dashboard’s request, usage and charge records correlated with them exactly.

Four calls on one installation show that the chain connects: a request goes in, a provider does some work, usage comes back, a charge lands and the dashboard shows each of those. They don’t qualify every combination of provider, model and transport, and they say nothing about how a real provider fails halfway through. The service’s qualification plan is built to keep that difference visible. An execution receipt has to name its provider, model and transport, its calls and retries, and its usage and cost where they’re available. A WebSocket connection isn’t counted as a provider call, and its handshake id isn’t treated as a correlation for individual messages. A missing terminal or an uncertain settlement stops the batch, and an outcome nobody could determine stays recorded as unknown, which is the rule the billing record follows one level down.

So when a call is interrupted, the figure on the statement comes from usage the provider reported, from a retained estimate marked as unknown usage because the provider may have worked, or from a refund because it can’t have. If the ledger can’t confirm the line, the operation waits for a person. The caller’s broken connection decides none of that.

Chris Chabot, November 2026