We Spent Three Months Chasing a Better Embedding Model. The Fix Took an Afternoon.

Day 18: Reranking — the fix everyone skips because they're too busy blaming the embedding model

Your retrieval pipeline passes every test you wrote for it. Users still say search feels dumb. Nine times out of ten, the fix isn't a better embedding model — it's a stage most teams never bother wiring in: a cross-encoder reranker.

Three months of re-embedding vs. one afternoon of reranking

I've watched this play out on more than one team, almost beat for beat. Month 1: relevance complaints land, someone declares 'our embeddings must be weak,' and the team goes off to benchmark a bigger embedding model. Month 2: they migrate the vector index, re-embed the entire corpus, and retune chunk sizes because someone read a blog post claiming 512 tokens is too big. Month 3: relevance has moved a little, complaints haven't stopped, and the team is now arguing about which embedding model to try next. Then someone bolts a cross-encoder reranker onto the existing bi-encoder retriever — no re-embedding, no reindexing, no migration — and relevance jumps more in one afternoon than the previous three months combined. The retriever was never the bottleneck. The ranking was.

What your bi-encoder is actually handing back

Here's what it looks like in practice — a query against a support-docs index, a standard bi-encoder (an OpenAI or Cohere embedding model, say) scoring by cosine similarity, no reranking involved. The numbers below are illustrative, not a benchmark I ran — but the pattern is the one I keep seeing in real pipelines, over and over:

Query: "how do I reset MFA on a locked account"

Top-5 from bi-encoder retrieval (cosine similarity):
1. 0.81  "Password reset policy overview and expiration rules"
2. 0.80  "Account lockout thresholds and admin unlock procedure"
3. 0.79  "Setting up multi-factor authentication for new users"
4. 0.77  "Resetting MFA when a user loses their authenticator device"  <- the actual answer
5. 0.76  "Security settings overview for enterprise admins"

The right chunk is sitting right there. At rank 4, wedged between three near-misses that sound plausible enough to fool a cosine score. If your app only feeds the top-1 or top-3 to the LLM — and most do — the one chunk that actually solves the user's problem never makes it into context.

Why bi-encoders do this — it's a designed tradeoff, not a bug

A bi-encoder embeds the query and every document independently, into the same vector space, ahead of time. At query time, it's just comparing vectors with cosine similarity. That independence is exactly what makes it fast enough to search millions of documents in milliseconds — your documents are already embedded, and the query gets embedded once. But that same independence is the limitation. The model never sees the query and the document side by side. It can't reason about whether a document actually answers the question — only whether two pieces of text are talking about similar things. A chunk about setting up MFA and a chunk about resetting MFA after losing a device sit right next to each other in embedding space, because they're topically adjacent, even though only one of them answers the question someone actually asked. Near-misses outrank exact answers because 'topically similar' and 'answers the question' are two different signals, and a bi-encoder can only see the first one.

The cross-encoder: a second stage, not a bigger first stage

A cross-encoder reranker — Cohere Rerank, BGE-reranker, a fine-tuned MiniLM cross-encoder, or an LLM prompted to act as a reranker — takes the query and each candidate document, concatenates them into a single input, and scores the pair jointly. Because the model is attending across both texts at once, it can actually judge whether a document answers the question, not just whether it's on the same topic. The catch is that this joint scoring can't be precomputed. You can't embed a query in advance because you don't know it yet, and you can't precompute every query-doc pair for your whole corpus because there are too many of them. So a cross-encoder only runs at query time, and only on a small set of candidates — which is exactly why it sits after the bi-encoder, not instead of it. Retrieval does the cheap job of narrowing millions of documents down to 50-100 candidates. Reranking does the expensive job of scoring just those candidates precisely.

Same query, after reranking

Same example, one stage added: take the top 50 candidates from the bi-encoder pass and run them through a cross-encoder reranker. Retrieval doesn't change — only the ranking does:

Query: "how do I reset MFA on a locked account"

Before (bi-encoder only):          After (+ cross-encoder rerank):
1. Password reset policy           1. Resetting MFA when a user loses
2. Account lockout thresholds         their authenticator device  <-
3. MFA setup for new users         2. Account lockout thresholds and
4. Resetting MFA (lost device) <-     admin unlock procedure
5. Security settings overview      3. MFA setup for new users
                                    4. Password reset policy overview
                                    5. Security settings overview

Nothing about the embeddings, the chunking, or the index changed. The reranker just re-sorted the same 50 candidates by 'does this answer the query' instead of 'is this in the same neighborhood as the query.' In an agentic RAG setup, that's the whole difference between your agent grounding its answer in the correct procedure and your agent confidently citing the wrong-but-adjacent doc.

The tradeoff: reranking is a second model call, and it costs you

Cross-encoders are slower per item than the bi-encoder's cosine math, full stop — you're running a transformer forward pass on every query-doc pair in your candidate set, not a cheap vector lookup. Reranking 50-100 candidates adds real latency, often tens to low hundreds of milliseconds depending on the model and batch size, and real cost, on every single query. In an agent pipeline that calls retrieval more than once per task — planning, tool calls, sub-agent fan-out — that per-call cost compounds fast, and it's exactly the kind of tail-latency contributor that hides until you're staring at p99 numbers on a multi-agent trace wondering where the time went. None of this is a reason to skip reranking. It's a reason to budget for it on purpose instead of discovering it in production. We'll size that budget — how many candidates to rerank, when to skip it on latency-sensitive paths — on Day 19 and 20.

When reranking won't save you

A reranker can only reorder what the retriever hands it. If the correct chunk never makes it into the top-50 or top-100 candidates, the cross-encoder never even sees it — and no amount of reranking fixes a recall problem. Before you reach for a reranker, check recall@k: for a sample of real queries with known correct answers, does the right chunk show up anywhere in the top-k candidates the retriever returns? If it's missing entirely, the problem is upstream — chunking split the answer awkwardly, the embedding model doesn't know the domain vocabulary, or the index isn't being queried the way you think. That's when touching chunking or embeddings is actually justified. Reranking fixes ordering problems. It does nothing for absence problems.

The five-minute check to run before anything else

Before you touch embeddings or chunk size again, pull the top-20 candidates for a handful of real, complained-about queries and just eyeball them. If the correct chunk is in that list but buried, you've got a reranking problem — add a cross-encoder stage and watch most of it disappear in an afternoon. If the correct chunk isn't in that list at all, you've got a recall problem, and that's the one that actually earns weeks of embedding or chunking work. Most teams skip this five-minute check and burn a quarter solving the wrong problem instead.

Extend your knowledge

• ▹Drop Cohere's Rerank API or an open-source cross-encoder like BAAI's BGE-reranker on top of your existing retriever. No reindexing, no re-embedding — it's a post-retrieval step you bolt on.
• ▹Build a small recall@k eval set — 20 to 30 real queries with known correct chunks — and check whether they land in your top-50 before you decide whether you're looking at a reranking problem or a retrieval problem.
• ▹Read up on two-stage retrieval (retrieve-then-rerank) as the standard architecture behind modern search and RAG systems. It's the same pattern production web search engines have used for years, adapted for LLM context pipelines.
• ▹Coming in Day 19/20: once reranking is in place, the next question is how many candidates to rerank and where it fits in your latency budget — especially in multi-agent pipelines making several retrieval calls per task.