My Hand-Tuned Hybrid Search Beat RRF — Until I Tested It on Queries I'd Never Seen

Day 12: Hybrid Search Fusion — Why Tuned Weights Lose to the Boring Default

Here's the part nobody puts in the hybrid search demo: fusion. It's five lines of code, it never breaks on stage, and it's the single easiest place to convince yourself you've made retrieval better when you've actually just memorized twenty queries.

The afternoon I spent tuning alpha weights

I had weighted-sum fusion running — score = alpha * dense_score + (1 - alpha) * bm25_score — and I burned an afternoon nudging alpha until my 20 test queries looked great. 0.73 felt like the number. Then I pointed the same setup at 30 queries I'd deliberately kept untouched, pulled from real support tickets I'd set aside weeks earlier. The 'placeholder' RRF baseline I hadn't bothered tuning beat my carefully tuned weighted sum on almost every one of them. I hadn't built a better fusion strategy. I'd built a strategy that had memorized 20 queries.

Quick recap: what 'hybrid' actually means

Hybrid search runs two retrievers over the same query — a sparse one (BM25, keyword and term overlap) and a dense one (embedding similarity) — then merges their two ranked lists into one. That merge step is fusion. It's what the rest of this lesson is about.

The setup: two legs, two fusion strategies, one held-out set

• ▹Corpus: a support-ticket knowledge base — exactly what a support agent or customer-facing RAG bot would query.
• ▹Sparse leg: BM25 over ticket text and resolution notes.
• ▹Dense leg: embedding similarity, same model I've used since Day 4, over the same documents.
• ▹Fusion strategy 1, weighted sum: normalize both score lists, then combine with a hand-tuned alpha.
• ▹Fusion strategy 2, RRF: skip the raw scores entirely and combine by rank position — 1 / (k + rank). One knob, k, and it barely matters.
• ▹The part I almost skipped: splitting my queries into 20 for tuning and 30 I wouldn't look at until the very end — a held-out set, labeled with which tickets were actually relevant.

The result: numbers don't lie, eyeballed queries do

Fusion strategy          | Recall@10 (20 tuned) | NDCG@10 (20 tuned) | Recall@10 (30 held-out) | NDCG@10 (30 held-out)
--------------------------|-----------------------|--------------------|--------------------------|------------------------
Weighted sum (alpha=0.73) | 0.91                  | 0.87               | 0.68                     | 0.61
RRF (k=60, untuned)       | 0.85                  | 0.80               | 0.79                     | 0.74

On the 20 queries I tuned against, weighted sum wins comfortably — of course it does, alpha was fit to exactly those queries. On the 30 it never saw, it falls apart, while RRF barely flinches. RRF isn't a better algorithm in some deep sense. It just had nothing to overfit.

Why this happens: you're fitting noise, not signal

Twenty hand-picked queries is a small, skewed sample. Slide alpha until those 20 look good and you're not learning anything general about your corpus — you're fitting the specific quirks of those 20: which ones happen to share exact keywords, which ones happen to be paraphrased. It's the same failure mode as picking a model checkpoint off five validation examples. The fact that alpha is one number instead of a million weights doesn't save you — a one-parameter model overfit to 20 points is still overfit.

The fix isn't a better formula

• ▹Build a real eval set first. Even 50 queries with labeled relevant documents beats 20 with none — you need enough held-out volume to actually catch overfitting when it happens.
• ▹Split it. Some queries for tuning, a disjoint set you don't touch until you're scoring a final candidate.
• ▹Default to RRF. There's no weight to overfit, it's brutally hard to beat with a small eval set, and it degrades gracefully when your two retrievers disagree.
• ▹Earn the right to hand-tune weighted sum — don't assume it. Once your eval set is big enough that a held-out split can actually catch overfitting, tune away. If you can't afford to hold out 30+ queries, you can't afford to tune alpha.

What this means for tomorrow's lesson

Fusion is step one of a longer pipeline, not the whole story — it just hands its merged list to whatever comes next. Tomorrow: what happens after fusion, when a cross-encoder reranker gets involved and changes the tuning problem all over again.

Extend your knowledge

• ▹Read the original RRF paper (Cormack, Clarke, Buettcher, SIGIR 2009) — it's short, and it explains exactly why rank-based combination shrugs off score-scale mismatches between retrievers.
• ▹Check how Elasticsearch and OpenSearch implement RRF natively for hybrid queries, and compare their default k to what you'd actually pick for your own corpus.
• ▹This week: build a 50-query labeled eval set for your own corpus, split it tuning/held-out, and rerun whatever fusion you're currently using against both halves.
• ▹Preview for Day 13: reranking with a cross-encoder. Think about why that changes what fusion even needs to optimize for — top-10 recall versus top-100 recall.