We Shipped a Vector Search 'Upgrade'. A Customer Searching an Exact SKU Got Nothing Back.

Day 11: Retrieval Eval — 'Semantic' Is a Claim, Not a Property

A cosine similarity score of 0.89 looks like proof. It isn't. It's proof of proximity — two vectors pointing in roughly the same direction — and proximity is not the same thing as relevance. Ship a vector search migration without an eval set measuring recall on real queries, and you cannot tell 'better retrieval' apart from 'different retrieval.' You find out which one you shipped from a support ticket.

The migration that shipped clean

We moved product search off BM25 (Elasticsearch) and onto a vector index — embeddings over product titles and descriptions, cosine similarity, top-k retrieval. Code review: clean. Smoke tests: green. A dozen manual spot-checks on fuzzy queries like 'wireless charger for desk' looked noticeably better than the old keyword index, so we called it a win and shipped. Two weeks later, support forwarded a ticket. A customer had searched an exact SKU — 'SKU-44819' — and gotten zero relevant results. The top hits were SKU-44812 and SKU-44820: semantically 'close,' and completely useless to someone reading a part number off an invoice. Nobody had tested an exact-match query, because nobody thought to. The eval, such as it existed, was vibes on fuzzy queries.

The seduction of cosine similarity

A similarity score feels like a quality signal because it's a number, and numbers feel objective. But cosine similarity only tells you two vectors point in roughly the same direction in embedding space — nothing about whether that direction has anything to do with what the user needed. A 0.89 between a query and a document means 'these sit close together in the model's learned geometry.' It does not mean 'this document answers the query.' Relevance is a judgment about task success; similarity is a geometric fact. Confuse the two and you get exactly what happened here: a migration that scores well on paper and regresses in production — the same trap as trusting an LLM-as-judge score or a leaderboard rank as ground truth for your use case, instead of checking it against what actually happened to the user.

The failure mode: when meaning IS the literal string

Embeddings are trained to generalize — to treat 'couch' and 'sofa' as neighbors, which is exactly what you want for a loose, natural-language query. But that same generalization becomes a liability the moment a query's meaning is inseparable from its exact characters. For those queries, blurring the string isn't a feature. It's the bug.

• ▹SKUs and part numbers — SKU-44819 and SKU-44812 differ by one character. To an embedding model they're neighbors. To someone quoting a part number off an invoice, they're different products
• ▹Error codes — ERR_502 and ERR_503 look 'close' to a model. To the engineer debugging a system, they're unrelated failure conditions
• ▹Proper nouns and IDs — a customer name, an order number, a ticket ID. The user wants the exact record, not 'the closest thing we have'
• ▹Version strings and model IDs — v2.1.3 vs v2.1.4 is a one-character diff with a real behavioral difference sitting behind it

Why BM25 never had this problem

BM25 is a keyword-frequency ranking function — it scores documents on literal term overlap, weighted by how rare and how discriminative each term is. It has no concept of 'semantically close,' so it never blurs SKU-44819 into SKU-44820: the token matches or it doesn't. That's also exactly where it falls down — it can't handle 'couch' vs 'sofa' at all. The mistake here wasn't choosing vectors. It was treating a strictly newer index as a strictly better one. Vector search and BM25 solve different halves of the retrieval problem. Call the migration an upgrade instead of a trade-off, and that's how the regression slips through unmeasured.

The fix wasn't a fancier model

The fix was building the eval infrastructure we should have had before the migration — not a better embedding model. Three pieces:

• ▹A real eval set pulled from actual query logs — not queries you made up, the messy distribution people actually type, including SKUs, error codes, and typos, each one labeled with the documents that count as relevant
• ▹Hybrid search — run BM25 and vector search in parallel, fuse the results (reciprocal rank fusion is the standard, unglamorous way to combine two ranked lists), so an exact-match query still gets caught by BM25 even when the vector branch blurs it
• ▹Recall@k and precision@k, measured before and after, on that eval set — not a one-time check, a number you track every time the index, the model, or the chunking strategy changes

# eval harness sketch — run before every retrieval change lands
def recall_at_k(eval_set, search_fn, k=10):
    total_recall = 0
    for query, relevant_ids in eval_set:
        results = search_fn(query, k=k)
        retrieved_ids = {r.id for r in results}
        relevant_ids = set(relevant_ids)
        total_recall += len(retrieved_ids & relevant_ids) / len(relevant_ids)
    return total_recall / len(eval_set)

# eval_set built from query logs, e.g.:
# [("SKU-44819", ["doc_44819"]), ("wireless charger for desk", ["doc_112", "doc_998"]), ...]

baseline = recall_at_k(eval_set, bm25_search)
candidate = recall_at_k(eval_set, vector_search)
hybrid    = recall_at_k(eval_set, hybrid_search)
# ship the change only if hybrid >= baseline on EVERY query segment,
# not just on average

The reusable lesson for Day 11

'Semantic' is a claim a retrieval system makes about itself — it's not something you can verify by reading the architecture diagram. Any change to retrieval — a new embedding model, a new chunking strategy, a new reranker, a switch from keyword to vector or back — needs an eval set built from real queries with labeled relevant results. The same logic reaches into agent pipelines: if a tool-calling loop leans on a retrieval step for context, a silent recall regression there quietly degrades every downstream decision the agent makes. And because the agent still answers fluently, nobody notices — not until the outcome is wrong. Fluency isn't correctness. Not in embeddings, not in agent output.

Carry this into tomorrow: treat retrieval quality like any other regression-prone system — with tests, not with vibes.

Extend your knowledge

• ▹Read the reciprocal rank fusion (RRF) paper, or the OpenSearch/Elasticsearch hybrid search docs, to see how BM25 and vector scores get combined without normalizing two incompatible scoring scales
• ▹Look at RAGAS or a similar RAG-eval framework to see how recall/precision-style metrics get adapted for retrieval-augmented generation specifically
• ▹Pull 50–100 real queries from your own product's search logs this week and hand-label the relevant documents — that's the minimum viable eval set, and it's worth more than any embedding-model comparison you'll read this year
• ▹Check the MTEB (Massive Text Embedding Benchmark) leaderboard for context on embedding model quality, but treat it as a starting point, not a substitute for your own eval set