Your RAG Bot Isn't Hallucinating — It's Just Serving a Two-Week-Old Cache

Day 13: Your RAG System Is a Cache Wearing a Search UI

Here's a bet: most of the 'the AI is wrong' tickets sitting in your queue right now have nothing to do with the model. They're stale cache entries dressed up as hallucinations. Nobody calls it that, because 'cache' isn't the word on the vector-database pitch deck — but that's exactly what it is. And if you can't tell the two apart, you'll burn a week rewriting prompts to fix a bug that's actually sitting in your indexing job.

Here's how it plays out in practice. Someone asks your support bot a pricing question. It answers confidently — with a number that's wrong. First reaction, every time: 'the LLM hallucinated.' So someone reruns the query with a sharper prompt. Nothing. They swap in a bigger embedding model. Still wrong. They bolt on a reranker. Still wrong. Three days later, someone finally opens the actual source doc — and there it is. The pricing page was updated two weeks ago, and the ingestion job that's supposed to re-index changed docs quietly stopped running sometime before that, and nobody noticed. The model didn't invent anything. It read the text it was handed and answered correctly from it. The text itself was just two weeks stale.

The one-line floor: what RAG actually is

If you're just joining the series: RAG (Retrieval-Augmented Generation) means fetch relevant text, stuff it into the prompt, ask the model to answer using only that text. That's the whole trick. There's no reasoning over your private knowledge base baked into the weights, no magic — just retrieval feeding a prompt. Embeddings, vector databases, rerankers: all of it exists purely to make that one 'fetch relevant text' step better.

The reframe: it's a cache, and caches have one hard problem

Once you see RAG as fetch-then-stuff, the architecture snaps into focus: your vector index is a cache of your source documents, re-encoded for similarity search. And every cache shares the same hardest problem — invalidation. Not retrieval quality, not embedding dimension, not which similarity metric you picked. Just: knowing when a cached entry stops being true and needs refreshing. RAG systems inherit this problem wholesale, and most teams don't even notice they've inherited it, because nobody pitches a vector database by calling it a cache.

This bites harder with agents than with a single chatbot answer. An agent that calls a retrieval tool mid-task, chains that answer into a decision, then calls another tool — a stale chunk there doesn't just produce one wrong sentence that a human might catch. It silently corrupts every downstream step of the plan, and nobody's reading the intermediate reasoning closely enough to notice.

Why teams debug it wrong

When the answer is wrong, the model is the part everyone can see, so it soaks up the blame. Every 'fix' that follows ends up aimed at the wrong layer:

• ▹Prompt tuning — rewrite the system prompt to be 'more careful,' add 'only answer from the provided context.' Does nothing if the provided context is already wrong.
• ▹Reranking — bolt on a cross-encoder to reorder retrieved chunks by relevance. It can't fix relevant-but-stale; it just sorts the stale chunk more confidently to the top.
• ▹Bigger embeddings — swap in a fancier embedding model to 'retrieve better.' Improves the semantic match, does nothing about whether the matched chunk still reflects reality.
• ▹Model swaps — move to a stronger LLM. The model was already reasoning correctly over what it was handed; a smarter model just states the stale fact more fluently.

All four are reasoning-layer fixes aimed at a data-layer bug. It's the equivalent of debugging a stale Redis cache by upgrading your CPU.

The 3-question checklist: model problem or index problem?

• ▹Did the source document change recently? Check its last-modified date against your last successful ingestion run. If the doc moved and the index didn't, you have your answer.
• ▹Does the wrong answer match an older version of the doc, word for word? If you can find that exact wrong number or sentence in a previous revision — git history, a CMS version log, a cached page — the model retrieved and repeated correctly, just from the wrong snapshot.
• ▹Does re-running the same query pull different chunks each time? Nondeterministic retrieval usually points at an index problem too — inconsistent chunk boundaries, duplicate or conflicting entries from a partial re-index, or old and new embeddings sitting side by side after a half-finished migration.

If the answer to all three is 'no' — the doc is current, the wrong answer doesn't match any text you can find, and retrieval is stable — now you're actually looking at a model reasoning failure, and prompt or model-layer fixes are the right tool. That's the minority case. It's just the only case those fixes were ever going to fix.

What actually fixes it

Treat the retrieval pipeline like any other cache and design for invalidation on purpose, not as something you bolt on after the first incident:

• ▹Trigger re-embedding on doc change — hook ingestion into whatever already tells you the doc changed: a CMS webhook, a git commit, an S3 object-modified event. Don't rely on a nightly full re-crawl and hope.
• ▹Timestamp every chunk — store the source doc's last-modified time alongside the vector, not just when you ingested it, so 'indexed recently' and 'source is recent' aren't the same question by accident.
• ▹Keep chunk-to-source traceability — every retrieved chunk should carry a pointer back to the exact doc, version, and location it came from, so 'is this chunk stale' is a lookup, not a three-day investigation.
• ▹Add a staleness check to your eval loop — for agent systems especially, log which chunks fed which decisions, so a bad outcome traces back to a specific stale chunk instead of a shrug and 'the agent messed up.'

None of this touches the model. It's pipeline engineering — the boring, unglamorous work that makes retrieval trustworthy before a single token of generation ever happens.

Takeaway: debug the pipeline before the model. When a RAG answer is wrong, check freshness and retrieval determinism before you touch the prompt. Tomorrow we go one layer deeper — into chunking itself, and how the way you slice documents quietly decides what your RAG system can ever know, no matter how fresh the index is.

Extend your knowledge

• ▹Look at your own RAG ingestion job and answer: what actually triggers a re-embed — a schedule, a webhook, or nothing? That answer tells you your real staleness window.
• ▹Add a `source_last_modified` and `indexed_at` field to your chunk metadata if you don't have one, and log the gap between them at query time.
• ▹Read up on classic cache invalidation strategies (TTL, write-through, event-driven invalidation) — the same taxonomy maps directly onto 'when do I re-embed a chunk.'
• ▹Tomorrow's Day 14 lesson: chunking strategy — how the way you slice a document decides what your RAG system can ever retrieve, independent of freshness.