Counting input tokens when the answering provider can change
Why one usage field means different things across model providers, and how to normalize token counts without hiding missing or contradictory data.
Two model providers can report the same field on the same kind of request and mean different things by it. In August 2026, Anthropic’s input_tokens counted only the tokens after the last cache breakpoint, with cache reads and cache creation reported beside it, so the size of the prompt was the sum of the three. OpenAI’s input_tokens counted the whole prompt and reported cached tokens as a subset of it. Either convention is perfectly reasonable, and if you only ever talk to one provider you can hard-code it in a single line and never think about it again.
We couldn’t, because model traffic on our platform went through a gateway that could fall back to another model, and a fallback can cross provider families. A request could be sent to a model from one family and answered by a model from another, with the usage arriving in the second family’s convention. So the useful question to ask of a usage report is not which model you asked for but who answered.
These numbers are made up for illustration, but they show how quietly this goes wrong. Take a response reporting 1,000 input tokens, 500 read from cache and 300 written to it:
| If the convention is | Uncached input | Size of the prompt |
|---|---|---|
| Inclusive, so 1,000 is the whole prompt | 1,000 - 500 - 300 = 200 | 1,000 |
| Exclusive, so 1,000 is only the uncached part | 1,000 | 1,000 + 500 + 300 = 1,800 |
Apply the wrong row and nothing throws, nothing looks odd, and the result is off by 800 tokens. A safeguard watching the context window that reads 1,000 when the prompt is really 1,800 acts too late, one that reads 1,800 when the prompt is really 1,000 acts too early, and any analysis of what the cache is doing inherits the same error with no visible seam.
When I corrected our accounting in the stream adapter, which followed Pi’s contract, I started from one internal rule: input means uncached input, cache reads and writes sit beside it, and the size of the prompt is their sum. Everything else followed from getting reports into that shape without losing what the provider actually said.
Usage that arrived mid-stream was treated as provisional, and normalization happened at the terminal event, because that was the event carrying the provider that actually answered. Before any arithmetic, three raw observations were recorded: the count the provider reported, the provider that reported it and the convention that applied. The adapter used a small set of provider names to choose the exclusive convention, and everything else was treated as inclusive. An exclusive count was copied across, an inclusive count had the cache reads and writes subtracted from it, and both results were marked exact.
The interesting case is an inclusive report that contradicts itself. One of the tests uses 100 input tokens with 300 read from cache and 100 written, which would mean that 400 of the provider’s 100 tokens went through the cache. Subtraction gives -300. The tempting repair is to clamp that at zero, and it’s a bad one, because zero is a valid and rather flattering answer: it says every input token went through the cache. The comment in the code warns against clamping the difference to “a supposedly genuine zero.” Instead the count kept the raw 100 as a conservative figure, the record took a state whose name spells out the whole situation, inconsistent, inclusive, conservative, and figures derived from that report were flagged as estimates.
Normalizing also must not overwrite what it normalizes. The internal contract governed the runtime’s own arithmetic, including the size of the context, but the usage event it emitted still carried the provider’s number. In the 1,000-token example that event’s input count stayed at 1,000, and beside it traveled a small, versioned block of accounting metadata holding the normalized 200, the convention, the provider and the state. Cost got similar treatment: the figure was the one the provider reported, converted to micro-dollars and rounded, and never recomputed from a price table, so context safeguards could work from coherent counts without anyone having quietly rewritten the bill. The record also kept the model requested, the model that responded, the model round, the provider attempts and the fallback sequence, so a strange number had a trail back to whoever said it.
Missing data got its own label for the same reason. Older usage with no raw observation was marked unavailable, and so was usage where the only name to hand was the model gateway adapter’s, since an adapter isn’t a provider and its name says nothing about a convention. None of those became zero. The emitter, meanwhile, refused malformed accounting outright: an unknown state, a negative count, a missing normalized count or a malformed convention threw, and nothing was emitted. That looks at odds with the tolerance shown to providers and isn’t. A provider’s contradictory report is a fact about the world and worth recording as one, while a malformed record from our own code is a bug, and passing it along would only teach every consumer to expect garbage.
The same metadata went on the calls that are easiest to forget, such as history analysis and compaction, and on one-shot calls, and it survived the round trip through the platform’s event mapper.
The whole scheme depends on downstream code reading the state. A cache-rate chart or a model comparison that averages normalized counts without looking at it will blend exact figures with conservative ones, and if someone treats unavailable as zero it restores the very error all this exists to remove.
Chris Chabot, November 2026