We Swapped BM25 for a Vector DB in One Sprint — Nobody Noticed for Three Weeks

Day 16 — A Vector DB Is a Downstream Choice, Not an Upstream One

Every agentic pipeline eventually needs retrieval, and the reflex is always the same: reach for a vector DB before anyone stops to ask whether it actually beats what you've already got. Get that order wrong and it doesn't blow up loudly — it costs you for weeks, silently, before a single alarm goes off.

The incident

On one RAG pipeline feeding an agent's context, we swapped BM25 for a vector DB in a single sprint. No eval before, no eval after — 'RAG' had been sitting on the roadmap as a checkbox, and once embeddings were wired in, the ticket closed. Three weeks later, someone doing manual QA on agent answers flagged that the retrieved chunks had gotten worse: right topic, wrong document, occasionally the wrong product entirely. Recall@k had dropped off a cliff, and nothing in our dashboards would have caught it — we were tracking latency and uptime, not recall@k, because nobody had asked us to.

The fix wasn't reverting to the vector DB's predecessor. It was admitting we'd never tested whether the switch was the right call to begin with.

Name the reflex

'Add RAG' is muscle memory for anyone building agents in 2026. The agent needs outside knowledge, so the plan writes itself: pick an embedding model, pick a vector DB, wire up a retriever tool, ship it. The decision gets locked in at the architecture-diagram stage, before a single query has touched the corpus. Nobody stops to ask the one question that actually matters — does semantic search retrieve better results than what we had before, on our documents, for our queries?

That question matters more now than it did in 2018-era search, because agent pipelines are unusually good at hiding retrieval failures. A bad chunk doesn't crash anything — the LLM just quietly reasons over the wrong context and hands back a confident, plausible, wrong answer. There's no stack trace for 'the agent answered fluently from the wrong document.'

The actual benchmark

After the incident, we finally ran the comparison that should have come first: same corpus (internal product docs and support tickets), same 200 real user queries pulled straight from logs, three retrieval setups, tested head to head.

• ▹BM25 + cross-encoder reranker: recall@10 ≈ 0.86, p95 latency ≈ 45ms
• ▹pgvector (same corpus, off-the-shelf embedding model): recall@10 ≈ 0.71, p95 latency ≈ 60ms
• ▹Dedicated vector DB (HNSW index, same embeddings): recall@10 ≈ 0.73, p95 latency ≈ 35ms

The dedicated vector DB beat pgvector on speed, exactly as advertised — that's the job it's built for. But both embedding-based setups trailed BM25+reranker by 13-15 points of recall@10 on this corpus, and latency was basically a wash between them. The 'upgrade' was a regression, and the team paid for it with three weeks of degraded agent answers before anyone thought to measure.

Why keyword search won here

Embeddings didn't lose to BM25 because they're inferior technology. They lose when the corpus and the query shape don't match what embeddings are actually good at — and on this corpus, three things stacked against them.

• ▹Short queries: users typed 'error 4023 checkout' or 'invoice API rate limit', not full sentences — there's barely any semantic content for an embedding to grab onto, but plenty of exact-match signal a keyword index nails instantly
• ▹Exact terms and IDs: ticket numbers, SKU codes, API error codes, and model numbers need exact or near-exact matching — an embedding model will happily hand back the semantically 'closest' error code, which is useless when the user needs that specific one
• ▹Domain jargon: internal product names and support shorthand were thinly represented in the embedding model's training data, so chunks that merely sounded similar outscored the correct literal match

This is the same failure showing up more broadly in agent tool use: an agent firing a retrieval tool with a short, keyword-heavy query — a variable name, an error string, a ticket ID — is exactly the case where semantic search struggles. And that's not an edge case for agent-generated queries. It's the common shape.

The decision framework

Before you buy — or default into — a vector DB, measure these against your own corpus and query set. Not a public benchmark. Not the vendor's demo dataset.

• ▹Recall@k baseline: run BM25 (plus a reranker if you can afford one) first, and record recall@k and MRR — that's the number embeddings actually have to beat, not zero
• ▹Query shape: pull 100-200 real queries from logs or agent traces — short and keyword-heavy favors lexical search, long and paraphrased favors embeddings
• ▹Corpus size and change rate: a small, mostly-static corpus rarely justifies the operational overhead of a vector DB; a large or fast-changing one (new docs hourly) is where embeddings and ANN indexes start earning their keep
• ▹Latency budget: watch p95, not the average, especially if retrieval sits inside an agent loop that calls it repeatedly per task — added latency compounds across a multi-step run
• ▹What would flip the verdict: heavy paraphrasing, conceptual 'what's similar to X' questions, cross-lingual queries, or a corpus too large for lexical indexes to stay fast — any of these tips the scale toward embeddings

Day 16 takeaway

Retrieval architecture is an empirical question, not a vendor decision. 'Vector DB vs. pgvector vs. Elasticsearch' is the wrong first question — the real one is whether embeddings beat your current baseline at all, on your corpus, for your queries. Skip that step and you're not doing RAG. You're doing vibes with an API bill attached. Day 17 builds the eval harness this story skipped, so you never again find out about a recall@k regression from a QA pass three weeks too late.

Extend your knowledge

• ▹Run your own BM25 vs. embeddings comparison on a real corpus with something like `rank_bm25` or Elasticsearch's BM25 scoring, tested against your existing embedding model, before you touch a vector DB
• ▹Read the original BEIR benchmark paper — it's the standard reference for how wildly retrieval methods swing across domains, which is exactly the point this lesson is making
• ▹Look at how your own agent traces phrase retrieval-tool queries; if they read like IDs and error codes rather than full sentences, that's a strong prior toward keyword search
• ▹Preview Day 17: building the eval harness (recall@k, MRR, latency) that should exist before any retrieval architecture decision — not bolted on after