Your Partition Key Was Correct. Then an AI Agent Ran It 40x Harder.

Day 8: The Partition Key Was Right. The Traffic Distribution Wasn't.

Every sharding post-mortem gets filed under the same headline: 'we picked the wrong key.' Almost none of them actually were wrong. The failure that keeps showing up is a good key that nobody paired with a way to reroute traffic once the distribution moved — and in AI systems, where a single agent loop can fire off thousands of calls with no human pacing it, that move happens a lot faster than most teams have budgeted for.

The incident

Say you shard a multi-tenant vector store by tenant_id. Sensible call — you pulled it straight from real traffic data at launch, and it holds for a year. Then one customer flips on an agentic workflow: an AI SDR bot that re-embeds and re-queries the knowledge base on every lead touch, running unattended, 24/7. Nobody's pacing those calls the way a person clicking through a dashboard would. Within weeks, that tenant's shard is carrying 40x the QPS of every other shard combined. On-call gets paged for the same shard, night after night. The obvious move is to repartition — spread that tenant's data across more nodes. It backfires immediately: repartitioning means a full data migration, and nobody budgeted the downtime window it needs.

The autopsy

Nobody was careless here. tenant_id came from actual access-pattern logs, it spread traffic evenly across every tenant that existed at the time, and engineers who understood the tradeoffs signed off on it. Judged at launch, it was the right call, full stop. The failure isn't in the choice. It's in what never got built next to it.

The reframe: a partition key is a bet, not a fact

A partition key encodes an assumption: that the relative traffic shares across key values stay roughly stable. That's a bet on the future, not a property of your data model — and every bet has an expiry date. In an AI product, the thing that blows up the bet is rarely organic growth. It's a step function: a customer turns on an agent workflow, a feature starts auto-generating embeddings, a batch job starts fanning out an LLM call per row. None of that shows up in the access-pattern data you reviewed at launch, because it didn't exist yet.

Why 'just repartition' is the wrong reflex

Repartitioning treats distribution drift as a one-off anomaly you fix once and move on from. It isn't. It's a recurring condition — traffic shape will drift again, and in an AI-driven product a single tenant's request volume can jump by orders of magnitude overnight because someone flipped on automation, not because your user base grew. Repartitioning is also the single most expensive lever you have: it usually means a live data migration, dual-write and cutover complexity, and, as above, downtime nobody scoped for. Reaching for your most expensive fix every time the same class of problem shows up isn't a data failure. It's a process failure.

The fix: separate 'where data lives' from 'how a client finds it'

The missing piece is a thin routing layer between the client and the shards. Instead of a client computing hash(key) and going straight to a shard, it asks a router — and the router can be repointed without touching a single row of data. Three versions of this show up in practice:

• ▹Directory-based sharding — an explicit lookup table, key → shard, that you can just edit. Moving one hot tenant off a shard becomes a row update instead of a migration. Vitess's VSchema for MySQL is a working example.
• ▹Consistent hashing with virtual nodes (vnodes) — each physical shard owns many small hash-range slices instead of one big one, so rebalancing moves a handful of vnodes instead of re-hashing the whole ring. It's the core idea behind Dynamo-style stores and Cassandra.
• ▹A dedicated lookup/routing service — a small stateful service, often just a cache in front of a config store, that every client call passes through. 'Where does this key live' becomes a runtime query instead of a formula baked into the client.

You're already running this pattern for LLM inference, even if you've never called it sharding. Put a router in front of a pool of vLLM or Ray Serve replicas, or add a model router that decides which backend serves a given call, and the client never computes which GPU to hit — it asks a router. Data sharding without that same layer is a client hardcoding an IP address for one specific GPU.

Why almost nobody builds this at launch

Because it's pure overhead until the exact week you need it. A routing layer adds a network hop, one more moving part to operate, and a decision you have to make up front — directory table or consistent hashing, how many vnodes — for zero visible payoff while traffic is still even. That's a legitimate tradeoff, not laziness: most systems never hit the skew that justifies it. The mistake isn't skipping the routing layer at launch. It's skipping the decision to skip it. Make the call on purpose, write down why, and know in advance what signal tells you you were wrong.

The heuristic: watch for this before the pager does

Two numbers will tell you it's time to add the routing layer, ideally before your on-call rotation tells you first:

• ▹Skew ratio — hottest shard's load divided by the median shard's load, measured in QPS or tokens/sec, never row count (an AI tenant can be tiny in storage and enormous in call volume). Past 3-5x, you're in the warning zone. 40x is the incident you just had.
• ▹Growth rate vs. shard count — if one key's traffic is growing faster than you're willing to add shards, uniform hashing alone can't bail you out, because it assumes growth stays spread evenly across keys.
• ▹In an agent-heavy system specifically, track per-tenant or per-workflow QPS/token throughput as a first-class metric — it's the earliest signal you'll get that an automation spike is turning into a hot-shard incident.

Where this leaves the course

Today assumed you already know how to choose a partition key — the industry has largely got that part right. What's usually missing is the layer that makes the choice reversible. Tomorrow picks up exactly there: what you build around the key so a distribution shift becomes a config change instead of a migration project.

Extend your knowledge

• ▹Read the consistent-hashing-with-vnodes section of the original Dynamo paper (Amazon, 2007) — still the clearest explanation anywhere of why vnodes make rebalancing cheap.
• ▹Look at Vitess's VSchema for a production example of directory-based sharding running on top of MySQL.
• ▹Study how a router in front of vLLM or Ray Serve replicas directs requests across inference pools — same indirection principle, applied to GPU capacity instead of data shards.
• ▹Before you need it: put per-shard QPS/throughput dashboards on your current shard map, so skew ratio is something you check, not something you discover from a page.