The retriever was innocent — the RAG bug was three days old and nobody was watching

Day 13 — RAG's real failure mode isn't retrieval, it's staleness

An agent cites a document that no longer exists, quotes it clause and all, sounds completely sure of itself. First instinct: blame the retriever. Wrong target. Nobody ever defined how fresh the index is supposed to be relative to the source of truth — and that undefined gap is the actual bug sitting in production, not a bad embedding or a sloppy chunk.

The deleted-document citation

Here's the timeline from an incident I worked through. A policy document gets indexed into the RAG store on Monday. It gets deleted from the source system — the CMS — on Thursday, because the process it described had changed. The following week, an internal agent answering a compliance question cites that document by name, quotes a clause from it, and builds its answer on a policy that no longer exists. The person on the receiving end goes to verify it in the source system. It's not there. What followed wasn't really a conversation about one wrong answer — it was the much harder question: if the agent can cite something that doesn't exist anymore, what else is it citing that's quietly wrong and nobody's checked?

Why the usual suspects were innocent

The first debugging pass went where everyone's does: retrieval quality. Every single check came back clean.

• ▹Chunking: boundaries were sensible, the cited clause sat intact and coherent inside its chunk — nothing stitched together wrong.
• ▹Embedding model: re-ran the identical query against the identical index, same top-k results came back, cosine similarity scores were exactly what you'd expect from a genuinely relevant match.
• ▹Reranker: ranked that chunk highly — correctly, because on paper it was a great match for the query. Rerankers score relevance, not existence.
• ▹Prompt: no hallucination, no misquote. The agent quoted the retrieved text faithfully and cited its source faithfully. It did exactly what it was supposed to do with the context it was handed.

Every component worked correctly, in isolation, on the input it received. The input was the problem — a chunk from a document deleted three days after it got indexed and never pulled from the vector store.

The actual mechanism: staleness as a race condition

This was never a retrieval-quality issue. It was a race condition between two systems that had never been synchronized: the source of truth (the CMS, where a document's real lifecycle happens — create, edit, delete) and the reindex job (a nightly or weekly batch that pushes changes into the vector store). That reindex job only handled creates and updates. A delete in the source system produced no event anywhere the reindex job was listening for — so the vector store kept serving a chunk for a document that, as far as the source system was concerned, was gone. The index wasn't wrong when it was built. It went wrong the moment the source diverged from it, and nothing was watching for the divergence.

The reframe: RAG is a replication system, not a search feature

Distributed-systems engineers already have the right mental model for this, they just apply it to databases: the moment you copy data from a source of truth into a second store that gets read independently, you've built a replica — and every replica has a consistency model, whether you designed one or not. A vector index built from your docs, your codebase, or your ticket system is a replica. Most teams build the retrieval path once, as an ETL job — pull docs, chunk, embed, load — and never come back to answer the question a replication system is actually supposed to answer: how far behind the source can this replica be, and does the caller know how far behind it currently is? Skip that, and you don't get eventual consistency. You get undefined consistency, which is worse, because nobody's accounting for it.

What we changed: freshness as a checkable, returned property

The fix wasn't a smarter retriever. It was making staleness visible instead of assumed. Two changes. First, the reindex job started consuming delete and update events directly from the source system instead of only doing periodic full or incremental adds, so the replica's lifecycle actually tracked the source's lifecycle. Second, every chunk returned by retrieval now carries a last_synced_at timestamp and a source_status check, so the caller — the agent, or whatever's generating the final answer — can reason about freshness instead of silently trusting it.

{
  "chunk_id": "policy-142-c3",
  "text": "Employees may expense up to...",
  "source_doc_id": "policy-142",
  "source_url": "cms://policies/142",
  "last_synced_at": "2026-09-15T02:11:00Z",
  "source_status": "unknown"  // set by a live check, not assumed "current"
}

That source_status field means the retrieval layer either confirms the doc still exists in the source system — cheap for a small number of top-k hits, a batched existence check, not a full re-crawl — or flags it explicitly as unverified. Then the agent prompt can carry an instruction: don't state something as current fact if source_status isn't confirmed within the last N hours, hedge instead. The bug was never retrieval quality. It was freshness living as an invisible assumption instead of a value you could actually inspect.

Why this gets worse with multiple agents

One agent reading a stale index produces one wrong answer, and a human can catch it. Put multiple agents on the same index — a planner, a code-review agent, a compliance-check agent, all pulling from the same RAG store — and a stale chunk stops being a single wrong answer. It gets picked up, restated, and reasoned over by every downstream agent that trusts the first one's output, and the staleness compounds silently across the fleet instead of surfacing once. That's a setup for a later lesson in this series on multi-agent RAG. For now, just register this: the consistency model you skip designing today is the same one that turns into a much harder distributed problem the moment more than one agent reads from the store.

Takeaway for today

Before you touch your retriever, your chunker, or your reranker, ask one question: how do I know this chunk is still true right now? If you can't answer that in a single sentence — pointing at an actual timestamp, event stream, or live check — that's the real bug in your RAG pipeline, and no amount of retrieval tuning is going to fix it.

Extend your knowledge

• ▹Read the consistency-models chapters in Martin Kleppmann's 'Designing Data-Intensive Applications' — the replication and consistency framing maps directly onto RAG index design, even though it predates LLMs.
• ▹Look at change data capture (CDC) tools (e.g. Debezium) as a pattern for propagating deletes and updates from a source system into a reindex pipeline in near-real-time instead of via periodic batch jobs.
• ▹Check what your vector database actually gives you for free: Pinecone and Weaviate both support arbitrary per-record metadata, and Weaviate tracks creation/update timestamps on objects automatically. pgvector is just a Postgres extension for the vector column — any timestamp or freshness field comes from your own table schema, not from pgvector itself. Don't assume the field exists until you've checked.
• ▹If you're building a coding-agent RAG pipeline over a codebase, the same problem shows up as indexing a branch or commit that's since been force-pushed or rebased — worth auditing your own pipeline for a 'last_synced_at' equivalent before it bites you.