Your RAG Pipeline Already Found the Answer — It Just Buried It at #7

Why this matters

Your pipeline is retrieving the right chunk more often than you'd guess — it's just parking it at rank #7 while the LLM only ever reads the top 3 to 5. That's not a retrieval failure. It's a ranking failure. And it has a fix cheap enough to bolt on this afternoon.

The hook: it was right there

Take a support-bot query like 'How do I reset my API key after a compromise?' Pull the top-10 chunks your vector store hands back and actually read them. Nine times out of ten, the chunk with the real revoke-and-rotate steps is in there somewhere — sitting at position 6 or 7, buried under five chunks that are all vaguely 'about API keys': pricing pages, key-creation docs, rate-limit docs. Your top-3 context window never sees it. The model didn't miss the answer. You never showed it the answer.

Diagnose: similarity is not relevance

Cosine similarity answers exactly one question: do these two pieces of text live in the same neighborhood of concept-space? It does not answer: does this chunk actually resolve this question? A bi-encoder embeds the query and every chunk independently, then compares vectors — query and chunk never get to interact. So a chunk that's merely topically adjacent (mentions 'API key,' 'security,' 'account') can outscore the one chunk that's operationally correct, because embeddings flatten away exactly the fine-grained, token-level interaction that tells you whether a pair actually resolves to an answer. It's the same failure mode you see when an agent's tool-retrieval step grabs a plausible-sounding tool instead of the right one — topical match, not functional match.

Name the fix: two-stage retrieval

The fix isn't a prompt tweak. It's architectural: split retrieval into two stages, with two different models doing two different jobs.

• ▹Stage 1 — bi-encoder (what you already have): embed query and corpus separately, run ANN search, optimize for recall across millions of chunks in milliseconds. Cast a wide net — pull the top 50, not the top 5.
• ▹Stage 2 — cross-encoder reranker: feed the query and each candidate chunk into the model TOGETHER, as one input. Attention layers can now compare query tokens against chunk tokens directly, so the score reflects actual query-answer fit, not just topical closeness.
• ▹The tradeoff is exactly why you need both stages: a cross-encoder is far more accurate but you can't index it — you can't precompute a score for a query that doesn't exist yet, and scoring your whole corpus per query would crawl. So the cheap model shrinks millions of candidates down to 50, and the expensive model puts those 50 in the right order.

How-to: bolting a reranker onto a Day-10 pipeline

If your Day-10 pipeline looks like embed query → ANN search top-k → stuff into prompt, the change is exactly one extra hop. Widen the candidate set (top 50 instead of top 5), rerank it, then truncate to whatever actually fits your context budget.

import cohere

co = cohere.Client("COHERE_API_KEY")

# Stage 1: your existing bi-encoder retrieval, but widen the net
candidates = vector_store.search(query_embedding, top_k=50)
docs = [c.text for c in candidates]

# Stage 2: cross-encoder rerank on query+doc pairs
result = co.rerank(
    model="rerank-english-v3.0",
    query=query,
    documents=docs,
    top_n=5,
)

reranked_chunks = [docs[r.index] for r in result.results]
# feed reranked_chunks[:3-5] into the LLM prompt, in this order

No Cohere budget, or no external API allowed? Same pattern, fully local, with an open cross-encoder:

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

pairs = [[query, doc] for doc in docs]
scores = reranker.predict(pairs)

reranked = [doc for _, doc in sorted(zip(scores, docs), reverse=True)]

That's the entire change. Same vector store, same chunks, same LLM — you've just inserted a re-scoring step before deciding what the model gets to read. Of every change I've made to a RAG pipeline, this single hop has produced the biggest jump in answer quality — bigger than swapping embedding models, bigger than tuning chunk size.

Cost and latency: when it's worth it

A reranker adds a real hop — typically tens to low-hundreds of milliseconds for 50 candidates, plus either API cost (Cohere-style) or GPU time if you self-host. Weigh that against what you're protecting.

• ▹Worth it: knowledge bases with hundreds to millions of chunks, ambiguous or overlapping documents, anything where a wrong citation is embarrassing or costly — support bots, internal tools, legal/medical-adjacent RAG, agent tool-selection over large tool catalogs.
• ▹Skip it or make it optional: small corpora (a few dozen docs) where top-5 recall is already near-perfect and there's little to reorder; hard real-time chat where every 100ms is felt and the corpus is narrow enough that bi-encoder ranking is already fine; prototypes where you haven't yet proven retrieval is even the bottleneck.
• ▹Middle ground: rerank only when the bi-encoder's top score falls below a confidence threshold, or only for agent-facing retrieval — where a wrong context silently corrupts a multi-step plan — and skip it for latency-sensitive user-facing chat.

War story: the wrong-doc complaints

On a support-bot project, we kept getting the same complaint: 'the AI keeps citing the wrong doc.' Always plausible-sounding, always adjacent, never actually correct. We assumed the embedding model was the problem and spent a week benchmarking alternatives — marginal gains, same complaint. The actual bug: the correct doc was almost always sitting in the retrieved set, just not in the top 3 we fed the model. We added a cross-encoder rerank over the top-30 candidates and cut wrong-citation complaints sharply within the same week, with zero changes to embeddings, chunking, or the LLM itself. The lesson that stuck with me: before you touch the embedding model or the prompt, check whether the answer was already in your candidate set and got ranked out. If it was, that's a ranking bug, not a retrieval bug — and reranking is a much cheaper fix than re-architecting retrieval.

Where this fits in the arc

Today's fix is local: it reorders the candidate list you already have. It does nothing if the right chunk never made it into that set in the first place — that's a recall problem, not a ranking problem, and it comes from how you chunked the source docs or how the query was phrased. Tomorrow (Day 12) we go one layer earlier in the pipeline: chunking strategy and query rewriting, which decide whether the right answer is even in the pool a reranker gets to work with.

Extend your knowledge

• ▹Run Cohere's rerank API, or the open cross-encoder/ms-marco-MiniLM-L-6-v2 from sentence-transformers, over your own top-50 candidates and diff the ordering against what you're serving today.
• ▹Read the Sentence-BERT paper and docs (Reimers & Gurevych) — the source of the bi-encoder vs. cross-encoder framing — for the architectural reasoning behind why joint encoding scores relevance better.
• ▹Look at BEIR benchmark results for retrieve-then-rerank pipelines to see how much rerankers actually move nDCG/recall across different domains.
• ▹Next lesson (Day 12): chunking strategy and query rewriting — the stage that decides whether the right chunk is even in the candidate pool for a reranker to work with.