Work out cache hit savings for llm instantly with clear inputs, formula shown and shareable results.
A cache turns the blended cost per request into a weighted average of hit and miss costs. Savings scale linearly with hit rate and with the gap between the two costs, so both prompt caching, where a cached prefix is billed at roughly a tenth of the input rate, and full response caching, where a hit costs almost nothing, pay off quickly at high volume.
Blended cost with cache
cost = requests x (hit rate x cost per hit + (1 - hit rate) x cost per miss); savings = uncached cost - cached cost
Exact-match response caching on open-ended chat rarely exceeds 10 to 20 percent, but prompt-prefix caching of a long system prompt or shared document context routinely exceeds 80 percent.
Stale answers when underlying data changes, and cross-tenant leakage if the cache key omits the user or permission scope. Always include authorisation context in the key.