Embeddings

Day 26: Embeddings, and why close in meaning doesn't make it the right file

Your coding agent just shipped a clean, tested fix to a file nobody has called in months. The model wasn't dumb. The retrieval step did exactly what it was built to do: it found the text most similar to the task. You'll run into this in every RAG pitch and in plenty of coding-agent setups, and people nearly always blame the model. The real problem is usually that a structural question was handed to a tool that only measures similarity.

The hook: a clean fix in the wrong file

We hit this with an internal coding agent. The task was a payment-retry bug. The agent searched the codebase, found the retry logic, wrote a neat fix and added a test. The test passed and the diff looked tidy. The bug was still in production.

The agent had edited the old retry module. That module had been replaced months earlier and was only waiting to be deleted. The test passed because it exercised the old module, and nothing in production called that module anymore. CI missed it, and so did the agent. The reviewer caught it only because they happened to remember the file was supposed to be gone.

The uncomfortable twist: nothing malfunctioned

When we traced it, the vector search had done its job. Honestly, the deprecated module was the chunk most similar to the task description. It had the same function names, the same comments about backoff and the same domain vocabulary. The retrieval got similarity right. We'd just asked it the wrong question. That raises the obvious follow-up: what does 'similar' actually mean here?

The concept in one picture: an embedding is a coordinate

An embedding model takes a chunk of text or code and returns a fixed-length list of numbers, usually hundreds or thousands of them. Treat that list as a coordinate on a 'map of meaning'. The model is trained so that chunks about similar things end up near each other. Search works in two steps: embed the query, then return the chunks with the nearest coordinates, most often measured by cosine similarity.

Picture a city map. Ask for the house nearest '12 Retry Street' and you'll get houses in the same neighbourhood, which helps when you don't know the exact address. The house next door is still the wrong house, though. Being nearby tells you two things are alike. It can't tell you who lives there, whether anyone still does, or which house the mail actually goes to.

# The whole idea, minus the model and the index
import numpy as np

def cosine(a, b):
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

query = embed("fix payment retry backoff bug")   # -> e.g. a 1024-dim vector
chunks = {
    "payments/retry_v2.py":        embed(open("payments/retry_v2.py").read()),
    "payments/legacy_retry.py":    embed(open("payments/legacy_retry.py").read()),  # deprecated
    "tests/test_legacy_retry.py":  embed(open("tests/test_legacy_retry.py").read()),
}

ranked = sorted(chunks, key=lambda k: cosine(query, chunks[k]), reverse=True)
# Nothing in this ranking knows which file is imported, called, or scheduled for deletion.
# It only knows which text is 'about' retries.

Why it works so well for documents

In documents, a question and its answer are often written in different words. A user types 'how do I get my money back?' and the policy page says 'refund eligibility'. Keyword search misses that completely. Embeddings close the vocabulary gap because both phrases land in the same neighbourhood of meaning. That's a large part of why RAG took off (see the earlier day on RAG and context): you can pull passages that answer the question even when they share no words with it, and then put them in the model's context.

• ▹Good fit: support docs, wikis, design notes, tickets and meeting notes, where the question is 'find me something about this'.
• ▹Also good: early discovery in a codebase you don't know yet. 'Where does this repo handle rate limiting?'
• ▹Weak fit: any question with exactly one correct answer that depends on how things link to each other.

Why it misfires on code

In practice, three close-but-wrong pairs kept showing up:

• ▹Test file and implementation. A test repeats the function names, the domain words and often the docstrings of the code under test, so the two sit side by side on the map. The agent edits the test, or treats the test's assumptions as the real behaviour.
• ▹Deprecated module and its replacement. In our case the replacement was usually written by copying the old module and changing it, which left the two almost identical in meaning. Only one is still in use.
• ▹Interface and mock. A mock has the same method names as the interface it fakes. Search 'how does PaymentGateway charge a card' and you may get back the fake that always returns success.

All three fail the same way. Embeddings capture what code is about. They don't capture what calls it, what replaced it, or whether it's still alive, and that is exactly the information an agent needs before it edits anything.

The deeper point: code is already structured

Prose has no built-in structure, so a rough map of meaning beats having no map. Code isn't like that. It already contains exact links: import statements, call sites, type hierarchies, the build graph, ownership files and git history. 'Is this module used?' has a definite yes or no answer. When you embed the code, those links get flattened into fuzzy closeness. Then you spend compute and retrieval tricks (re-ranking, metadata filters, hybrid scoring) trying to roughly rebuild links you had exactly from the start.

Embeddings also go stale. An index built last week has no idea a module was deprecated yesterday. grep and git always read the code as it is right now.

What we changed

The agent still uses embeddings, but only for the first step: getting a rough sense of where to look. Before it edits anything, it now has to confirm the target with structural tools:

• ▹grep or ripgrep for the exact symbol, to find every definition and call site.
• ▹Check the call graph or imports. Can a production entry point reach this file, or do only tests reach it?
• ▹Read the recent git history. Has the file changed lately, or did some commit replace it? Look for 'deprecate', 'replace' or 'migrate' in the log.
• ▹Check paths and names. tests/, mocks/, legacy/, _old and __fixtures__ are warning signs. Don't treat them as targets.

# Embedding search said: payments/legacy_retry.py. Verify before editing.
rg -n "from payments.legacy_retry|import legacy_retry" --glob '!tests/**'   # any live importers?
rg -n "def retry_with_backoff" payments/                                     # all definitions
git log --oneline -n 10 -- payments/legacy_retry.py                          # recent life signs
git log --oneline -S "legacy_retry" -n 5                                     # who moved away from it

Some frontier coding agents ended up in a similar place. Claude Code, for example, relies mainly on agentic search. The model runs grep, glob and file reads in a loop and decides what to look at next, instead of depending on a pre-built vector index of the repo. The Claude Code team has said publicly that this worked better for them than RAG over embeddings. Not every tool went this way, and some still lean on embedding indexes. Part of why agentic search works is the point above: it reads exact symbols, and it always sees the current code.

A rule you can keep

Embeddings answer 'what is like this?'. Structure answers 'what is connected to this?'. Before you trust a retrieval result, whether you're a person or an agent, work out which of those two questions you're really asking. If the result is going to be edited, deleted or treated as the source of truth, you're asking a connection question. Let embeddings find the candidates and let structure make the call.

• ▹Pitfall: treating the top similarity hit as the answer. It's a candidate and nothing more.
• ▹Pitfall: blaming the model for a bad edit when the retrieval step gave it the wrong file.
• ▹Pitfall: indexing tests, mocks and legacy code at the same weight as live code, with no metadata to tell them apart.
• ▹Pitfall: a stale index. If the code changes every day and you re-index once a week, you're working from an outdated map.

Tomorrow, Day 27: chunking. Before anything gets embedded, someone decides where to cut the text, and that decision limits what can ever be found.

Extend your knowledge

• ▹Try it yourself. Take a few files from your own repo (an implementation, its test, a mock and a legacy copy), embed them with any embedding model and rank them against a task description. You'll see how close the wrong ones get.
• ▹Read the original Retrieval-Augmented Generation paper (Lewis et al., 2020). RAG was designed for knowledge-intensive NLP tasks over text. Editing code wasn't the problem it set out to solve.
• ▹Look at how code-intelligence tools like Sourcegraph, or the Language Server Protocol, provide go-to-definition and find-references. Those are the exact links that embeddings blur.
• ▹Watch a coding agent such as Claude Code work through a real task, and count how often it reaches for grep, glob and file reads instead of semantic search.