The 95% Cache Hit Ratio That's Quietly Burning Your Origin

Day 4: Cache Hit Ratio — One Number, Two Definitions

Everyone in your org has looked at a 95% cache hit ratio and relaxed. Almost nobody has asked which machine produced that number, or what it politely leaves out.

Two people, two truths, one dashboard

Your infra lead is telling you origin load is climbing and users are complaining about latency. Your CDN rep is pointing at a dashboard that says 95% hit ratio, everything's fine. Neither one is lying to you. They're just reading two different instruments that happen to share a label.

What 'cache hit ratio' actually means

Cache hit ratio is the share of requests a cache layer can answer without doing the full work the request would otherwise trigger — hits over hits-plus-misses. That definition is correct. It's also incomplete, because 'the full work' means a different thing depending on which layer you ask.

The split: edge 'hit' vs origin 'miss'

At the edge, a hit means the CDN node answered on its own — no contact with your origin, zero cost to you. Clean win. But there's a middle case every CDN happily files under 'hit': the 304 Not Modified. The edge asks your origin, 'still fresh?' Your origin says, 'yep, unchanged.' The edge serves the cached copy and the dashboard ticks up a hit. Meanwhile your origin just ran a full request-response round trip, executed your app logic to answer the freshness check, and burned real CPU and latency doing it. On the origin side, that's a miss wearing a hit's costume.

• ▹True edge hit — the request never leaves the CDN; your origin sees nothing at all.
• ▹304 revalidation — edge asks origin 'still valid?', origin computes the answer, edge serves the cached body. The CDN counts this as a hit.
• ▹True miss — origin computes and returns a full new response, which the edge then caches.

Why this isn't pedantry

A team that only watches the vendor's hit-rate graph will drift toward optimizing for more 304s, because 304s look free — small payload, graph still says 'hit.' But a 304 is still a round trip to origin. Still a request your app has to authenticate, route, and validate. Set your cache-control headers too aggressively toward revalidation — short max-age, no immutable, stale-while-revalidate left unused — and you can push hit ratio past 95% while origin CPU and tail latency both climb. The graph goes up. The user's experience gets slower. Both of those are true at the same time, and the dashboard only shows you one.

The tell: where the two numbers diverge

Here's where you'd actually catch it: as a gap between two views of the same traffic. The CDN dashboard sits near-flat and high. Your origin access logs, filtered down to 304 status codes, show a steadily rising share of total origin traffic. Origin p95 latency creeps upward even though the vendor's summary makes request volume at origin look low — because that summary counts 304s as handled, not as origin load. Nobody pulls origin logs directly, and this gap just sits there, invisible. The only artifact anyone looks at is the vendor's number, and that number will never tell on itself.

The fix: ask one question, pull one metric

• ▹Ask your CDN vendor directly: does your hit ratio count 304 revalidations as hits, and can you break out true-hit from revalidation-hit separately? Most vendors can split this the moment you ask — they just don't surface it by default.
• ▹Pull it yourself from origin logs: requests per second (or per minute) broken out by status code, specifically 304 vs. 200 vs. everything else. That's the number that actually tracks what your infra pays for, regardless of how the CDN labels it.
• ▹Compare trend lines, not snapshots. If vendor hit-ratio is flat or rising while your origin 304-rate is also rising, you're paying more origin cost for the exact same 'win' on paper.

Framed for the AI era: the same trick, new layers

This exact ambiguity now lives inside your LLM stack too, not just your CDN. Anthropic's prompt caching reports a cache-read on the prompt prefix — a genuine cost and latency win, close to a true edge hit. But inference servers like vLLM report KV-cache / prefix-cache hit rates that can look great while a partial-prefix match still triggers a partial recompute — the 'hit' still burned real GPU time, just less of it, exactly like a 304 still burns a round trip. In a multi-agent pipeline, an orchestrator's 'cache hit' on a tool result or retrieved context can mean 'we skipped the sub-agent call entirely,' or it can mean 'we called the sub-agent anyway just to re-validate the cached result was still fresh' — same word, same trap. If you're scaling an agent fleet and someone tells you '90% cache hit rate on retrieval,' ask them the exact question from above: hit against what, and does it still touch the expensive layer underneath.

Bridge to tomorrow

Once you can tell a true hit apart from a revalidation, the next question waiting for you is: how do you make sure a cached entry actually gets thrown out when it's supposed to? That's cache invalidation — quoted as one of the two hard problems in computer science for good reason.

Extend your knowledge

• ▹Read RFC 9110's conditional-requests section (ETag, Last-Modified, If-None-Match) alongside RFC 9111's validation model (HTTP Caching) — it spells out exactly what a 304 obligates your origin to do.
• ▹Go check your own CDN's docs — Cloudflare, Fastly, Akamai, whoever — for how they define 'hit ratio' and whether they expose a revalidation-hit breakdown. The definitions differ vendor to vendor, more than you'd expect.
• ▹If you run LLM inference, pull up vLLM's prefix caching / automatic prefix cache metrics and hold them against your actual GPU utilization graph, the same way you'd hold CDN hit-rate against origin logs.
• ▹Pull one week of your own origin access logs and compute the 304 share yourself before your next vendor QBR. Bring the number, not the feeling.