Your Embeddings Are Fine. Your Vector DB Is Fine. So Why Is Retrieval Still Wrong?

Why This Matters

Here's the thing nobody wants to hear when their retrieval quality is bad: it's probably not your database. If you've got a 'correct' embedding model and a 'correct' vector store and retrieval still feels off, go look upstream — at the boundary you drew through the document before either of those systems ever touched the text. A chunk that slices a fact away from its context is gone. Not degraded, not deprioritized — unretrievable. You can re-rank it, you can migrate to a fancier DB, you can throw a bigger embedding model at it. None of that puts the sentence back together.

The pattern I keep seeing

Teams will burn weeks on the vector DB bake-off — Pinecone vs. Weaviate vs. pgvector, latency charts, cost-per-million-vectors spreadsheets — and then spend maybe ten minutes deciding how the document gets cut into chunks. Usually they just take whatever the framework defaults to: 500 tokens, 1000 tokens, whatever. Then retrieval comes back mediocre, and the reflex is to blame the embedding model, or go swap the DB again. That's optimizing the wrong knob. A vector database is a search index over whatever units of text you fed it. Feed it broken units, and it will index that brokenness with perfect fidelity.

Name the actual bug

A vector database does exactly one job: store vectors, return the nearest ones to a query vector. That's it. It has zero opinion on whether a given chunk actually makes sense standalone. If your splitter chopped a sentence, a table row, or a qualifying clause in two, the database doesn't know or care — it'll embed both halves, search both halves, and hand back whichever half sits closer to the query. Often that's the half without the answer in it. What looks like 'vector search is bad' is, almost always, a chunking bug in a trench coat.

A concrete example: the severed fact

Say your support doc has this line: 'Refunds are processed within 5-7 business days, except for annual plans, which take 10-14 business days due to proration.' Run that through a naive fixed-size splitter — 60 tokens, no overlap — and the cut can land right after '5-7 business days.' Now you've got two chunks:

Chunk A: "...Refunds are processed within 5-7 business days,"
Chunk B: "except for annual plans, which take 10-14 business days due to proration..."

A user asks: 'How long does a refund take for my annual plan?' The query embedding lands closest to Chunk A — it's literally about refund timing — so that's what gets pulled. The LLM answers '5-7 business days' with complete confidence, because as far as it can see, Chunk A is the whole, authoritative answer. It never even knows Chunk B exists. Nobody hallucinated. Nothing malfunctioned. The chunk boundary manufactured a wrong answer before retrieval ever ran.

The reframe: get the decision order right

In a RAG stack, these three decisions aren't peers you can tune independently — they're a strict dependency chain. Get the order wrong and you'll spend your time tuning stages two and three against chunks that were broken from the start.

• ▹1. Chunk boundary — decide what counts as one complete, self-contained unit of meaning (a full policy clause, a full Q&A pair, a full code function) before anything else
• ▹2. Embedding model — chosen to represent those units well; a better model can't compensate for a unit that's semantically incomplete
• ▹3. Vector DB — chosen for scale, latency, and filtering needs; it only matters once what it's indexing is actually meaningful

A diagnostic you can run today

You don't need an eval framework for this. Pull 5 real user queries from your logs — real ones, not test questions you made up — and for each, go find the sentence(s) in the source doc that actually answer it. Then just check: did your splitter keep that answer, whole, inside one chunk?

• ▹If the answer spans two chunks, that's a severed-fact bug — no amount of top-k tuning fixes it
• ▹If the answer is intact but buried in a chunk full of unrelated content, embedding quality is diluted — that's a granularity problem, not a model problem
• ▹If the answer is intact and well-isolated but still isn't retrieved, now you have an embedding or ranking problem worth investigating
• ▹Do this for 5 queries before you touch your vector DB config — it takes 20 minutes and tells you exactly which layer is actually broken

Where this goes next

This is why chunking strategy isn't a preprocessing footnote you set once and forget about — it's the load-bearing architecture decision in RAG. Tomorrow's lesson on retrieval evaluation is, underneath the metrics, measuring exactly this: whether the units you retrieve actually contain the answers your users are asking for.

Extend your knowledge

• ▹Read Greg Kamradt's '5 Levels of Text Splitting' — a widely-referenced walkthrough of chunking strategies from naive fixed-size to fully semantic/agentic splitting
• ▹Look at LlamaIndex's node parser / chunking documentation to see structure-aware splitters (sentence-window, hierarchical) as alternatives to fixed-size splitting
• ▹Look at Pinecone's Learning Center articles on chunking strategies for RAG — practical tradeoffs between chunk size, overlap, and retrieval quality
• ▹Run the 5-query diagnostic from this lesson against your own current RAG pipeline before tomorrow's lesson on retrieval evaluation