Cache-Hit Rates Are Easy to Get Wrong. We Found Three Different Formulas in the Wild — Including Our Own Stack.
Prompt-cache reads bill at roughly a tenth of full input price on the major providers. That single pricing fact makes the cache-hit rate one of the most consequential numbers in agent cost work: it decides whether a multi-turn agent is cheap or expensive, and it’s the first number anyone checks when a bill looks wrong.
It’s also a number with no agreed definition. Auditing cost tooling — including our own — we found three different formulas in production use:
1. Reads over total input (correct). cache_read_tokens / total_input_tokens, where total input = fresh + cache reads + cache writes. This answers the question people are actually asking: what share of my input was served at the discount?
2. Writes counted as hits (overstates). Cache creation tokens bill at a premium (~1.25×), not a discount — a write is a miss that populates the cache. Formulas that add writes to the numerator report a warm cache on turn one, when you’re paying the most. We found this variant in one of our own dashboard views.
3. The double-counted denominator (understates). If your metering stores prompt tokens as total input — which most does, matching what providers bill — then computing cached / (promptTokens + cached) counts the cached tokens twice in the denominator. This roughly halves the reported rate: a real 96% renders as about 50%.
The worked example
That third formula was live in our own MCP connector, and it produced a lesson worth more than the bug: I saw “~50% cache hit” on my own coding-agent traffic, asked a frontier model to diagnose it with me, and got a plausible answer — tool-result bloat dragging the ratio down. It fit the priors. We started sketching a fix.
Then we tested the claim against raw runs instead of the dashboard number. The real rate was ~96%. There was no bloat problem; runs with more tool-result bytes actually cached better, because long runs warm the prefix cache. Every step of the reasoning downstream of the number had been sound. The number was the only thing wrong — and it was wrong in a way that produced confident, well-scoped engineering work aimed at a problem that didn’t exist.
That’s the property worth internalizing about metrics in anything self-optimizing, human or agent: a wrong metric doesn’t just mislead — it manufactures work. An agent reading that connector would have run the same play without the step where we got suspicious: “optimize” the pipeline, watch a number that was never going to move, continue. The loop closes on garbage. Quesma made an adjacent point well — the average run is lying to you — about aggregates hiding individual behavior; the formula version is harsher because the aggregate isn’t hiding truth, it’s fabricating a falsehood.
The fix took one line, plus a rule we now hold: one definition (reads over total input), asserted everywhere the number appears, and the same verification we apply to external claims applied to our own dashboards. Numbers that drive agent behavior are a safety surface. It’s also why AI agent cost monitoring starts with getting the definitions right.
The checklist
If you consume a cache-hit rate from any tool — ours or anyone’s:
- Ask whether writes count as hits. They shouldn’t.
- Ask what the denominator is, and whether cached tokens are already inside it.
- Sanity-check against pricing: a long-running coding agent with a stable prompt should sit well above 90%. If your dashboard says 50%, suspect the formula before the agent.
- If a surprising number is about to drive real work, verify it against raw usage data first. The answer is occasionally a feature you don’t need to build.