We Added a Cache Node. Eight Minutes Later the Database Caught Fire.

Day 16 — Consistent Hashing Doesn't Promise a Calm Migration

Every tutorial on consistent hashing stops the second it proves "only 1/N of keys move." Nobody tells you what happens in the ten seconds right after those keys move and right before the new node has anything cached — and that gap is exactly where the outage lives, especially in front of an LLM inference fleet or an agent-facing cache.

The 2AM page

The change ticket said "routine": add a node to the Redis ring sitting in front of primary Postgres — more cache capacity, less load on origin. Eight minutes after the deploy, origin CPU pinned at 100% and every read-path service started timing out. The ring had strictly more capacity than before. The database was on fire anyway.

The assumption that was half true

Everyone on the team could recite the consistent-hashing pitch on command: add the Nth node to a ring of N, and you only remap roughly 1/N of keys — not the whole keyspace, the way naive mod-N hashing would. That part's true, and it's the entire reason consistent hashing exists. Mod-N hashing means every single node add or remove reshuffles almost everything.

• ▹Consistent hashing guarantees: on average, ~1/N of keys move when you add or remove a node — this is a statement about *how many* keys move.
• ▹It says nothing about *when* they move, *how fast* traffic hits them, or *what backs the new node's slice of keyspace* the instant it joins the ring.
• ▹The team's mental model conflated "few keys move" with "low disruption." Those are different claims — one is about volume, the other is about traffic pattern over time.

What actually happened on the ring

When the new node joined, it inherited a contiguous slice of hash space from its neighbor — everything between hash(C) and hash(N), say. Every key that used to route to C and happened to land in that arc now routed to N instead. N had just booted. Its cache was empty, full stop. Every request into that arc was a guaranteed miss, and because real traffic doesn't arrive one key at a time, dozens to hundreds of concurrent requests for the same now-cold keys landed on the origin DB simultaneously — not spread out, not gradual, all in the same few seconds the node flipped live.

This is a cache stampede, but the trigger isn't a hot key expiring — it's a cold *range* getting switched on. Whatever system the ring exists to protect takes the full, undamped weight of that range the second the topology changes.

Why virtual nodes made it worse, not better

Virtual nodes (vnodes) solve a different problem entirely: with one ring position per physical node, load distribution across nodes is lumpy, and a new node's slice is one big contiguous arc — easy to reason about, easy to pre-warm. We had vnodes configured, a common production pattern (Cassandra's classic default was 256 per node for years, before newer versions dropped it to 16 and made it tunable), specifically to smooth that lumpiness out.

• ▹More vnodes for the new node meant its share of keyspace was carved into dozens of small, scattered arcs instead of one contiguous block.
• ▹Instead of one predictable cold zone hitting origin from one direction, we got a stampede from many small cold ranges, arriving simultaneously from every part of the keyspace.
• ▹The exact smoothing vnodes give you for steady-state load distribution works against you the moment the topology changes — there's no single "the new range" to reason about or warm up as a unit. There are dozens.

The fix that shipped that night vs. the fix that shipped that month

The immediate patch had one job: stop concurrent identical misses from all reaching origin at once. That's request coalescing, also called single-flight — the first request for a cold key fetches from origin and populates the cache, and every concurrent request for that same key rides along on that one fetch instead of issuing its own. Paired with a rate limiter on cache-fill traffic per key range, it capped how much simultaneous cold-range traffic could actually reach the DB.

// immediate patch: coalesce concurrent misses, rate-limit fills
function get(key):
    if cache.has(key):
        return cache.get(key)

    // singleflight: only one in-flight origin fetch per key
    return inflight.getOrCreate(key, () => {
        rateLimiter.acquire(originFillBucket)
        value = origin.fetch(key)
        cache.set(key, value)
        return value
    })

// real fix: warm the new node's ranges BEFORE it's in the read path
function addNodeToRing(newNode):
    ranges = ring.previewRangesFor(newNode)   // vnode-aware, all scattered arcs
    for range in ranges:
        keys = origin.sampleHotKeys(range, topN=...)
        for key in keys:
            newNode.cache.set(key, origin.fetch(key))
    ring.commit(newNode)   // only now does it start receiving reads

The real fix — shipped over the following weeks, not that night — was pre-warming: compute the exact ranges (all of them, across every vnode) the new node will own before it ever joins the ring, pull the hot keys for those ranges from origin or from whichever node is currently serving them, populate the new node's cache, and only then flip it into the live ring. Adding capacity stopped being a config change. It became a migration with a plan, a rollback path, and a defined "done."

The rule this incident produced

Never add a node to a hash ring at read time without a warm-up phase. Treat every ring topology change — node add, node remove, rebalance — as a migration: define what gets pre-populated, how you verify it's populated, and what the blast radius on origin looks like if warm-up is incomplete or skipped entirely.

Why this bites harder in an AI-era stack

The physics are identical anywhere you route by key across a fleet, and an AI stack has more of these rings than most 2015-era architectures ever did. LLM inference gateways route requests by session or prefix hash to get KV-cache locality — it's what vLLM's and SGLang's prefix-cache-aware routing is actually doing under the hood. Add a GPU node to that routing ring cold, and you get the identical stampede, except "origin" is now a full-precision KV-cache recompute, which costs far more per request than a Postgres read ever did. Vector DB shards behind a RAG pipeline, per-agent session-affinity routers in a multi-agent fleet — same failure mode, every time. The ring math says "1/N keys move." The incident says "all of them move at once, uncached, straight into whatever's sitting behind the ring." If you're autoscaling an agent fleet or an inference cluster by adding nodes to a consistent-hash router under live traffic, warm-up isn't optional. It's the difference between the scale-out event relieving load and becoming the incident.

Where this connects to tomorrow

Once you accept that ring changes are migrations and not config edits, the next question is what actually triggers one. You don't want to pre-warm and migrate on every minor load blip, and you don't want to wait until a node is already melting to notice. Day 17 covers the signals that tell you a ring is imbalanced enough to justify a change — how to catch it while it's still a planning problem, not a page.

Extend your knowledge

• ▹Read the original Amazon Dynamo paper's section on consistent hashing and vnodes — it's the source of the "more vnodes for load balance" trade-off this incident exposed the dark side of.
• ▹Look at how vLLM or SGLang implement prefix-cache-aware request routing for LLM inference — it's a related consistent-hashing-for-locality pattern, with a GPU KV-cache recompute standing in for the origin DB.
• ▹Study Go's singleflight package (or an equivalent in your stack) as a reference implementation of request coalescing — it's the exact primitive used for the immediate patch here.
• ▹Check how your current ring/cache client (Redis Cluster, Cassandra, a custom consistent-hash router) exposes "what range will node X own" before a topology change — that API is the prerequisite for building a pre-warm step.