We Swapped in a Frontier Embedding Model. Recall Didn't Move. Here's What Was Actually Broken.

Day 8: Retrieval Plateaued? Stop Blaming the Embedding Model

Here's the reflex every team has: retrieval quality stalls, so you reach for a bigger, pricier embedding model. Nine times out of ten that's the wrong move. An embedding model can only compress the text you hand it — if your chunker already sliced the answer in half before it ever got encoded, no amount of extra parameters brings it back. This lesson is how you tell which problem you actually have.

You upgraded the model. Nothing moved. Now what?

Say you swap in a bigger, more expensive embedding model — same eval set, same queries, same top-k — and recall barely twitches. Don't read that as a failed experiment. Read it as a diagnosis: the bottleneck isn't how much the model can represent, it's what you're feeding it to represent. The next move isn't another model swap. It's opening the actual chunks your pipeline retrieved and reading them like a human would.

Go read the failing chunk. Raw.

Grab one query your eval set scores badly on. Pull the actual top-k chunks your pipeline retrieved — not the source doc, the exact text that got embedded. Nine times out of ten, the problem is staring at you within seconds.

Query: "What's the retry backoff for the payments webhook?"

Top-1 retrieved chunk (512-token split, no overlap):
"...configurable per integration. See table below for
recommended values."

[END OF CHUNK — table starts in the NEXT chunk, never retrieved]

Top-2 retrieved chunk (mid-table cut):
| Retries | Delay  |
|---------|--------|
| 3       | 1s     |
[CHUNK ENDS HERE — remaining rows, including "webhook: 5, 30s",
are in a different chunk with a different embedding, never surfaced]

The chunk that got embedded and indexed literally doesn't contain the answer. It contains a pointer to the answer — "see table below" — with the table surgically removed. No embedding model, however large, can encode information that was never in the text it was given.

Embeddings compress. They don't resurrect.

An embedding model's whole job is to take the text sitting in front of it and turn it into a vector that preserves semantic signal. That's the entire contract. It has no access to the rest of the document, no memory of what came before the chunk boundary, no way to reconstruct a definition that got severed from the term it defines. If the boundary already destroyed the signal, the model is just compressing noise — very faithfully.

Re-embed that same amputated chunk with a small model, a mid-tier model, and a frontier model — you'll get three different vectors, all equally wrong, because all three are encoding a sentence fragment that stopped containing the fact your query needs. That's the tell: if quality doesn't budge across model sizes, the ceiling was set before the model ever saw the text.

The 10-minute diagnostic

Before you touch the model config again, run this against your worst-performing queries:

• ▹Pull the 5-10 queries your eval scores worst on.
• ▹For each one, print the actual top-k chunks that got retrieved — the exact text that was embedded, not the source document.
• ▹Read every chunk in isolation, with zero surrounding context, and ask yourself: does this chunk alone contain enough to answer the question, or at least support the answer?
• ▹Watch for the usual suspects — a sentence cut mid-clause, a table or code block sliced across a chunk boundary, a pronoun or 'as shown above' pointing at something outside the chunk, a heading orphaned from the section it titles.
• ▹If most of your worst queries show one of these, you don't have a model problem. You have a chunking problem.

Chunking rules that actually move the needle

• ▹Split on structure, not token count. Headers, list items, table boundaries, code block boundaries — a chunk boundary should land on a seam the document already has.
• ▹Never split a table mid-row or a code block mid-function. Keep the smallest self-contained structural unit whole, even if that blows past your target chunk size.
• ▹Size your overlap to preserve referents, not to hit a percentage. If a chunk opens with 'this configuration' or 'the table above,' the thing it's pointing at needs to live inside that chunk — via overlap, or by re-including the heading.
• ▹Treat 'chunk size: 512' as a hypothesis to test, not a default you inherit. It's a number someone else picked for a document shape that isn't yours — a knowledge base of short FAQ entries and a doc full of nested tables need different boundaries.

# structure-aware, not token-count-aware
def chunk_document(doc):
    chunks = []
    for section in doc.sections:               # split on headers first
        for block in section.blocks:            # then on structural units
            if block.type in ("table", "code"):
                chunks.append(with_context(section.heading, block))
            else:
                chunks.extend(
                    split_by_paragraph(block, overlap="preserve_referents")
                )
    return chunks

Where this sits in the course

Day 7 gave you the embedding — the mechanism that turns text into something you can compare. Day 8 is the step that happens before that: chunking decides what text the embedding model is even allowed to look at, which means it decides what the embedding is allowed to represent. Day 9 picks up retrieval tuning — reranking, hybrid search, query rewriting — and all of it assumes your chunks already carry the signal. Tune retrieval on broken chunks and you're just optimizing around a hole.

Before you upgrade the model again

Pull ten chunks your pipeline actually retrieved for real queries. Read them out loud, one at a time, with nothing else in front of you. If you can't answer the question from the chunk alone, no embedding model can either. That's the whole diagnostic — and it costs you ten minutes, not a model migration.

Extend your knowledge

• ▹Swap your token splitter for a structure-aware one — LlamaIndex's node parsers or LangChain's MarkdownHeaderTextSplitter both cut on document structure instead of a fixed token count.
• ▹Read Anthropic's contextual retrieval writeup — it prepends surrounding context to each chunk before embedding, which is a direct fix for the orphaned-reference problem above.
• ▹Look at unstructured.io if you're parsing PDFs or HTML — it extracts structural elements (headings, tables, code) before you ever have to decide where a chunk boundary goes.
• ▹Do the diagnostic for real this week. Pull ten chunks from your worst-scoring eval queries and read them raw before you touch a single model setting.