RAG (Retrieval-Augmented Generation)

Day 27: RAG (Retrieval-Augmented Generation)

Your internal docs bot will eventually quote a real policy, link to the right section, and still give the wrong answer. Nothing is broken when that happens. That's RAG doing exactly what it was built to do. If your team is building a chatbot on internal docs, you're building RAG, whatever the slide deck calls it. Learn where it actually breaks and you won't spend weeks tuning the wrong part.

The confidently cited wrong answer

A support engineer asks the internal assistant: 'What is the refund window for annual plans?' The bot replies: '30 days, per the Refund Policy, section 4.2.' There's a link. The link works, the document is real, and section 4.2 does say 30 days. The engineer passes it on to a customer.

Last quarter the policy changed to 14 days. Someone saved the new version in a different folder, and nobody removed the old one. The bot quoted a real document accurately and the answer was still wrong.

So the question for this lesson is: if retrieval worked and the model didn't hallucinate, what broke?

The concept in 60 seconds

Think of RAG as an open-book exam. The model doesn't answer from memory. It looks things up first, then answers from what it found.

• ▹Retrieve: search your documents for the passages that best match the question.
• ▹Augment: put those passages into the prompt, right next to the question.
• ▹Generate: the model writes its answer from those passages and, if things go well, cites them.

This ties back to two earlier days in the series. Embeddings are the usual way of measuring 'relevant'. The context window is the reason you only get to include the top few passages and not the whole wiki.

question
   │
   ▼
search the index  (embeddings, keywords, or both)
   │
   ▼
top-k passages     (usually somewhere between 3 and 10)
   │
   ▼
prompt = instructions + passages + question
   │
   ▼
LLM  →  answer + citations

Why teams reach for it

• ▹The model has never seen your private data: your HR policies, your API, your runbooks.
• ▹It doesn't know anything recent either, because training stops at a cutoff date.
• ▹Fine-tuning is slow and expensive, and it's a bad place to keep facts that change. Try 'un-training' last quarter's refund policy.
• ▹With RAG you change what the model knows by editing documents, not weights. Update a file and the next answer changes.

That last point is RAG's superpower and also its trap. If editing a document changes the answer, then a wrong, stale or abandoned document changes the answer too. And because there's a citation attached, the wrong answer sounds official.

Where we looked first (the wrong place)

When our prototype started giving wrong answers in production, we did what most teams do and went after the retrieval pipeline:

• ▹Chunk size: smaller chunks for precise facts, bigger ones for context. Some question types got better and others got worse.
• ▹Chunk overlap: fewer answers sliced in half at a chunk boundary. A small win.
• ▹A reranker (a cross-encoder that re-scores the top results): the best passage landed first more often. This one was a genuine improvement.
• ▹Hybrid search (keyword search such as BM25 alongside vector search): this helped with product codes, error strings and acronyms, which embeddings handle badly.

Every one of these nudged our eval set in the right direction, and I'd make all of them again. Not one of them fixed the refund answer, because retrieval wasn't what failed. The old policy really was the passage most similar to the question. The retriever did its job. It just had no way of knowing the document was out of date.

The real failure mode: stale truth

A retriever ranks text by how well it matches the question. It can't tell which document is current, approved or still true. Two versions of a policy look almost the same to a similarity search, and the older one can easily come out on top. Maybe its wording is closer to how people phrase the question. Maybe it's been copied into more pages over the years. Similarity isn't validity.

In practice, stale truth tends to turn up in three ways:

• ▹Superseded policies: v3 of the travel policy is live, but v1 and v2 are still indexed and still match the question.
• ▹Deprecated API docs: the bot recommends an endpoint that was removed two releases ago. Put a coding agent in the loop and you get broken code instead of a wrong sentence.
• ▹Ownerless wiki pages: the 2021 'How deploys work' page that nobody owns or updates, and everyone is too nervous to delete. Your people learned to ignore it. Your retriever never did.

The fix is an org fix: treat the corpus like code

We already know how to keep a large, fast-changing body of text trustworthy, because we do it every day with source code: ownership, versioning, review and CI. The knowledge base needs the same discipline.

• ▹1. Every source has a named owner, meaning a person or a team. 'Confluence' doesn't count. If nobody will own a source, it doesn't get indexed.
• ▹2. Every document has a 'valid until' or 'review by' date. Expired documents are excluded from retrieval, or at the very least flagged in the answer ('this source is past its review date').
• ▹3. Versioning, so the newest version wins. When v2 is published, v1 gets archived and removed from the index. Leaving it sitting next to v2 isn't enough.
• ▹4. CI checks for knowledge, run on a schedule the way you run tests. Look for broken links, documents with no owner, documents past their review date, and contradictions, where two sources give different answers to the same question.

# frontmatter on every indexed document
title: Refund Policy
owner: team-billing
version: 2
supersedes: refund-policy-v1
status: active          # active | archived
review_by: 2027-01-31

# filter at retrieval time, before ranking matters
def eligible(doc, today):
    return (
        doc.status == "active"
        and doc.owner is not None
        and doc.review_by >= today
    )

candidates = [d for d in search(query, k=50) if eligible(d, today)]
passages = rerank(query, candidates)[:5]

That code is the easy part. The hard part is deciding who fills in 'owner' and who is allowed to archive v1, and those are organisational calls. An ML engineer can build the filter. They can't make Billing own the refund policy, and they can't give anyone permission to delete the 2021 deploy page. An EM or tech lead can. That's why stale truth is usually a management problem that looks like an ML problem.

A quick gut-check for your own RAG

Five yes/no questions you can answer on Monday:

• ▹Can you name who owns each source in the index?
• ▹Do you know which document in the index is oldest, and whether it's still true?
• ▹When two sources disagree, does the bot notice and say so, or does it quietly pick one?
• ▹When a doc is deprecated, does it disappear from answers within a day?
• ▹Does your eval set include questions whose answers have changed, so you'd catch the bot giving the old one?

Three or more 'no' answers? Fix ownership and expiry before you touch chunk size.

Where this is heading

In agentic systems, retrieval is turning into a tool the agent chooses to call, instead of a fixed step that runs before every answer. The agent decides when to search, what to search for and whether to search again. You get more flexibility, and stale truth gets harder to catch. When several agents read the same corpus, one outdated page can travel through planning, coding and review in a single run, and each agent's output reads like confirmation to the next one. That problem is where tomorrow's concept starts.

Takeaway

• ▹RAG means look it up, then answer.
• ▹Retrieval finds text that looks similar. It doesn't find text that's true.
• ▹Give every document an owner before you tune anything else.

Extend your knowledge

• ▹Read the original paper, Lewis et al., 'Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks' (2020), to see what the term meant before the vendor pitches got hold of it.
• ▹Anthropic's engineering post 'Introducing Contextual Retrieval' (2024) is a clear, practical walk through chunking, hybrid search and reranking. Those are the knobs to turn once ownership is sorted.
• ▹Try an evaluation framework such as Ragas and add 'changed answer' cases: questions whose correct answer changed recently, where giving the old answer counts as a failure.
• ▹Do an audit this week. Export your index's document list with last-modified dates and owners, sort oldest first, and go through the 20 oldest with the teams they belong to.