Your Cache-Hit Ratio Hit 95% — and Your Best Customers Still Think You're Slow

Day 12: Cache-Hit Ratio Is a Vanity Metric

Your cache-hit ratio can sit at 95% and your best customers can still think your product is broken. Both things are true at once, and the dashboard alone won't tell you which one you're looking at. A cache-hit ratio only reports on the majority — it says nothing about the minority stuck behind it, and in most real products that minority is exactly the traffic you should care about most: logged-in users, personalized pages, anything dynamic.

The contradiction nobody connects

Here's a scene that plays out more often than it should. Same quarter, two documents: a dashboard slide with a cache-hit ratio north of 95%, and a support queue quietly filling with 'the site feels slow' tickets. Nobody in the room puts these side by side, because nobody owns both. The CDN team owns the dashboard. Support owns the tickets. The gap between them belongs to no one — and that gap is exactly where a meaningful slice of your users are living.

Recap: what a CDN actually covers

Quick recap from Day 11: a CDN is your first scaling layer. It sits in front of origin, serves cacheable responses from the edge, and soaks up most of your traffic before it ever reaches your servers. Today is about the part it doesn't cover — the request that can't be served from the edge, and what happens to it once it isn't.

Why the hit-rate line goes up while user experience goes down

This isn't a paradox. It's arithmetic. A cache-hit ratio is a ratio, not a distribution — and ratios have a blind spot built into them. As a product grows, two things happen at once. Cacheable traffic (static assets, public pages, unauthenticated marketing routes) grows fast, because it scales with raw visitor count. Uncacheable traffic (personalized dashboards, search, checkout, anything behind auth) grows slower and gets a fraction of the engineering attention, because it 'only' shows up as the 5% miss. The ratio climbs because the numerator — hits — grows faster than the denominator. Nobody is watching the absolute latency of that 5%; they're watching it shrink as a percentage, which is a very different thing. An average, or a ratio, is a machine for hiding a tail. The long tail doesn't get slower because someone made a bad call. It gets slower by default, because it's the one path nobody is on the hook for keeping fast.

The war story: the routes that were quietly rotting

The pattern shows up again and again. The cache-hit dashboard is green, sometimes higher than last quarter, and the team quietly files 'CDN' under solved-problems and moves on. Then a customer — usually a paying one, usually on a personalized or logged-in view — files a ticket saying the app feels sluggish. Not down. Just sluggish. Support checks uptime: fine. Someone checks the CDN dashboard: fine. The ticket sits there looking like a one-off, until three or four more show up over the next few weeks, all from the same handful of routes — account dashboards, search-with-filters, anything carrying a session token. Those routes were never eligible for edge caching to begin with; they're user-specific by definition, so every single request takes the full round trip to origin. Pull a TTFB distribution sliced by route instead of blended across all traffic, and the picture flips: cached routes cluster in the low double-digit milliseconds, while personalized routes have a long right tail stretching into full seconds — often because origin is doing expensive per-request work (auth checks, personalization logic, sometimes a chain of internal API calls) that nobody ever put a latency budget on. The dashboard was never lying. It was just answering a question nobody should have mistaken for 'is the product fast.'

The fix isn't 'cache more' — it's owning the miss path as a system

You can't cache your way out of traffic that's inherently dynamic or personalized — and that's not a caching failure. That's caching doing exactly its job. The actual fix is treating the miss path as its own system, with its own owner and its own SLOs, the same way you already treat the cache layer:

• ▹Origin latency budget — set an explicit p95/p99 target for requests that hit origin, separate from any CDN metric, and alert on it independently.
• ▹Stale-while-revalidate — serve a slightly stale cached response instantly while refreshing it in the background, so 'dynamic-ish' content doesn't have to be fully synchronous on every request.
• ▹Edge-side includes / partial caching — cache the static shell of a personalized page (nav, layout, static content blocks) at the edge and stream only the truly personal fragment from origin, instead of treating the whole page as one all-or-nothing cache decision.
• ▹A p95-on-misses metric, published next to hit-rate on the same dashboard — so a healthy ratio can never again visually stand in for a healthy tail.

The architectural takeaway

A CDN isn't one system. It's two: the cache, and origin-on-miss. Real architecture means designing and owning both — not shipping the cache layer, watching the dashboard turn green, and calling it done while the origin path quietly degrades under load nobody is measuring. This is Goodhart's Law showing up in system design, in its most common shape: any metric that blends easy traffic and hard traffic into one number gets gamed by the easy traffic growing, while the hard traffic — the part your best customers actually depend on — rots in the blind spot the metric creates.

The same failure mode is playing out right now, one layer up, in AI infra — just wearing different names. LLM inference systems have their own version of cache-hit ratio: vLLM's prefix caching reports how much of the KV cache gets reused instead of recomputed, and Anthropic's prompt caching does the same thing for repeated prompt prefixes. Teams watch that number climb as the easy traffic — repeated system prompts, common tool schemas — grows, while the miss path (a novel prompt, a long multi-agent tool-call chain, an agent that wanders off the beaten branch) pays full prefill cost with no budget and no SLO of its own. In a multi-agent pipeline this gets worse, not just proportionally worse: one uncached hop early in the chain doesn't just add its own latency, it stalls every downstream agent waiting on it. The tail from a single cache miss can dominate end-to-end p99 even while the aggregate cache-hit number looks great on a slide. If you're building or operating agent fleets, the lesson transfers exactly: put a latency budget on the cold path, not just a hit-rate on the warm one.

The reframe

So the fix for tomorrow's dashboard: hit-rate doesn't get to stand alone anymore. Put p95 (or p99) latency on cache misses right next to it, sliced by route or request type — not blended into one comforting average. A hit-rate that climbs while miss-path p95 also climbs isn't a win. It's two systems moving in opposite directions under one green label. Day 13 picks up where the edge ends: how you actually scale the origin behind it — load balancing, so 'origin-on-miss' stops meaning one server hoping for the best.

Extend your knowledge

• ▹Pull your own CDN or reverse-proxy logs and compute TTFB p95 sliced by route, not blended — see if any personalized/dynamic route is hiding behind your headline hit-rate.
• ▹Read up on stale-while-revalidate (originally an HTTP Cache-Control directive, RFC 5861) and check whether your CDN/edge provider (Cloudflare, Fastly, CloudFront) supports it for your dynamic routes.
• ▹If you're running LLM inference, look at vLLM's or your provider's prompt/KV-cache hit-rate metrics and ask the same question: is there a p95-on-cache-miss number anywhere near it?
• ▹Add one alert this week: origin (or cache-miss) p95 latency, independent of your existing CDN hit-rate alert.