Why Your Six-Month-Old RAG System Is Quietly Getting Dumber

Day 15: Embedding Drift — The Bug That Isn't in Your Prompt, Retriever, or Reranker

Embedding models don't crash. They don't throw errors. They just quietly get worse at their one job while every dashboard you're staring at stays green. If you set yours up six months ago, chunked your docs, shipped it, and moved on — that's the one part of your RAG stack nobody's opened since, and it's already drifting under you.

The war story

A team I worked with ran a support-ticket RAG assistant that had been rock solid for a year. Then, over one quarter, agents started filing a new kind of complaint: 'the bot gave a confidently wrong answer.' The team did what everyone does — rewrote the system prompt three times, tuned the reranker's top-k and score threshold, swapped pure dense retrieval for hybrid BM25+dense, and eventually started eyeing the LLM itself ('maybe we just need a bigger model'). Every change produced a small bump in the metrics that decayed back to baseline within a week. Three sprints later, someone finally asked the question nobody had thought to ask: 'when did we last check whether the embeddings still make sense for what we're storing today?' Turns out the corpus had quietly shifted underneath them — a new product line had dumped in a wall of documents full of unfamiliar terminology, acronyms, and a writing style nothing like the original knowledge base. The embedding space hadn't moved. The data living inside it had.

Why embeddings are last on the suspect list (or not on it at all)

When retrieval quality degrades, most engineers — and most AI debugging agents, for that matter — work through a very predictable order:

• ▹1. The prompt — reword it, bolt on few-shot examples, tighten the instructions.
• ▹2. The retriever config — top-k, filters, hybrid search weights, MMR diversity settings.
• ▹3. The reranker — swap models, nudge the score cutoff.
• ▹4. The LLM — reach for a bigger or newer model, chalk it up to a reasoning failure.
• ▹5. (Rarely reached) The embeddings — filed away as a one-time infrastructure decision, not something that can quietly rot.

This order exists because everything above embeddings is cheap to iterate on — reword a prompt, redeploy, done in thirty seconds. Touching the embedding model feels like reopening a decision that's already settled, so it's the last thing anyone reaches for, if they reach for it at all. AI coding agents inherit the same blind spot: ask one to debug a RAG regression and it'll reach for prompt tweaks and retriever configs first, because that's where the debugging surface 'looks' biggest. Embeddings barely register as mutable state in most people's — or agents' — mental model of the pipeline.

What embedding drift actually is

An embedding model is trained on a snapshot of text — a fixed slice of vocabulary, topics, and document structure, frozen the moment training stops. It doesn't move again unless you retrain or swap it out. Your corpus, on the other hand, never sits still. New product lines, new jargon, new document shapes — going from prose to structured JSON, or from English-only to mixed-language content — all push your live data further from whatever distribution the model was trained on. The model hasn't changed. The gap between what it's good at representing and what you're now asking it to represent has.

The tell in that war story was specific and repeatable: near-duplicate queries — the same question asked slightly differently — started coming back with increasingly inconsistent, off-target top-k results, and the inconsistency tracked almost exactly with how recently the matching document had been added. Old documents still retrieved cleanly. New-vocabulary documents kept getting buried under older content that was lexically 'safer' but semantically beside the point. That's the fingerprint of drift, not a retriever bug — the embedding space simply doesn't separate the new concepts cleanly anymore.

The diagnostic that found it

The fix wasn't a new tool — it was isolating the one variable nobody had touched in a year. The team built a small golden set: roughly 50 real query/expected-document pairs, half from old content, half from the new product line. They ran recall@k and MRR against the live index, then re-ran the exact same golden set against embeddings generated from a snapshot of the original model config, on both cohorts. The split was stark: recall on old-content queries hadn't budged; recall on new-content queries had dropped by more than half. That's an embedding-layer regression, full stop — the retriever and reranker were doing their jobs correctly on embeddings that no longer represented the new content.

# Minimal drift probe: same golden set, isolate the embedding layer
for cohort in ["old_docs", "new_docs"]:
    golden = load_golden_set(cohort)
    embeddings = embed_model.encode(golden.corpus)   # current model, current corpus
    scores = eval_recall_at_k(embeddings, golden.queries, k=5)
    print(cohort, "recall@5:", scores.recall_at_k, "mrr:", scores.mrr)

# old_docs recall@5: 0.91  mrr: 0.84
# new_docs recall@5: 0.42  mrr: 0.31   <- the smoking gun

Why this stays invisible in most monitoring

Retrieval metrics almost always get built once, at launch, as a pass/fail gate: 'recall@5 needs to clear 0.8 to ship.' Nobody schedules that eval to re-run against fresh corpus samples, so there's no trend line — just one historical number everyone quietly assumes still holds. Decay that happens a few points a month looks exactly like noise on a dashboard that only shows point-in-time production signals — latency, error rate, thumbs up/down — with no standing golden-set eval underneath it. You have to deliberately re-measure against a fixed yardstick to see a slope instead of static.

Checklist: catching drift before it's an incident

• ▹Keep a golden set of 30–100 query/document pairs, and refresh it quarterly with recent content — not just what you sampled at launch.
• ▹Re-run recall@k / MRR against that golden set on a schedule (monthly is reasonable for most teams) and plot it as a trend, not a single number.
• ▹Segment results by document recency or source — if new content underperforms old content, that's drift, not noise.
• ▹Cheap early-warning signal: track the cosine-similarity distribution between newly ingested documents and your existing corpus centroid. A shift there precedes a visible drop in recall.
• ▹When you add a genuinely new content type or vertical — new product, new language, structured data instead of prose — treat it as a trigger to re-run the golden-set eval immediately, not wait for the next cycle.
• ▹Version and log which embedding model and checkpoint produced every vector in your store, so 'was it the embeddings' is a five-minute check, not an archaeology project.

Where this fits in the 30-day arc

Every earlier lesson in this series treated embeddings as a Day-1 decision: pick a model, chunk your docs, build the index, ship it. That framing is exactly what makes drift invisible — it files embeddings under 'done' instead of 'needs maintenance,' right next to your schema and your index. And this matters more now than it used to: agentic systems increasingly lean on embeddings for more than document retrieval — tool routing, cross-session memory recall, semantic caching of LLM calls — all of which decay the same silent way as your corpus and usage patterns evolve. Day 16 picks up here: once you've confirmed drift, what do you actually do about it — full re-embed, fine-tune, adopt a newer model, or run a staged dual-index migration — without a risky big-bang cutover.

Extend your knowledge

• ▹Set up a scheduled eval job — even a simple cron plus script — that re-runs your golden set weekly or monthly and logs recall@k/MRR to a time series you can actually plot.
• ▹Read the MTEB (Massive Text Embedding Benchmark) leaderboard methodology to understand how embedding models get benchmarked — useful context before you pick a replacement.
• ▹If you're on OpenAI, Cohere, or Voyage embeddings, check their model versioning and deprecation docs. These providers version embedding models explicitly by name (text-embedding-3-small vs. -large), so the real risk isn't a silent swap under an unchanged name — it's mixing vectors from two different model versions in the same index if you ever partially re-embed your corpus.
• ▹Preview Day 16: remediation strategies for confirmed drift — full re-embed vs. fine-tuning vs. staged dual-index migration.