Model the effective latency a cache delivers and the memory it needs.
Effective latency is dominated by misses once the hit rate is high, because the origin path is orders of magnitude slower. Raising a hit rate from 88 to 95 per cent nearly halves effective latency. This is why the last few percentage points of hit rate are worth pursuing even though they require disproportionate memory.
Cache Memory
Effective latency = cache latency × hit rate + origin latency × miss rate
Effective latency = cache latency × hit rate + origin latency × miss rate Effective latency is dominated by misses once the hit rate is high, because the origin path is orders of magnitude slower. Raising a hit rate from 88 to 95 per cent nearly halves effective latency.
This is why the last few percentage points of hit rate are worth pursuing even though they require disproportionate memory.
This calculator takes 5 inputs: Entries held in cache, Bytes per entry, Hit rate, Cache latency, Origin latency. The pre-filled defaults are a realistic starting point — replace them with figures from your own environment for a result you can act on.