Vector databases

Day 25: Vector Database

Your retrieval logs say every query succeeded, yet your agent keeps calling a function that was deleted eleven minutes ago. A vector database can return outdated context with a high similarity score and log no error at all. Vector databases show up in nearly every RAG pitch and agent architecture diagram, and teams often add one before deciding what it should retrieve. Once you know what it stores and how it finds things, you can tell whether it's helping your agent or feeding it stale context.

The hook: a deleted function, and logs that say everything worked

I keep coming back to this case. A coding agent refactors a module and deletes a helper called `formatInvoiceLegacy`. A few tasks later, the same agent asks its retrieval layer how invoices get formatted. It gets back a chunk that matches well, and that chunk still contains `formatInvoiceLegacy`. The agent writes a call to it and the build fails. Then it spends several turns fixing an import for a function that no longer exists.

The retrieval logs look fine. The query ran, the top-k results scored high on similarity, and latency was normal. You only see the problem in the timestamps. About eleven minutes passed between the edit that deleted the function and the retrieval that returned it. For those eleven minutes the index still described the old repo. Remember that gap. The rest of this lesson explains it.

The one-paragraph definition

An embedding model turns a piece of text into a list of numbers, usually a few hundred to a few thousand of them. Treat that list as a point in 'meaning space', where texts that mean similar things sit close together. A vector database stores millions of these points with the original text and some metadata. When you give it a query, it quickly returns the stored points nearest to the query's point. It stores meaning as coordinates and retrieves by distance. (Day-by-day readers: this is the storage and lookup layer under the embeddings and RAG ideas from earlier days. You don't need to relearn anything.)

A picture to keep: the library shelved by topic

Picture a library that shelves books by subject instead of alphabetically by title. Books on grief sit together whatever they're called, and so do books on distributed consensus. Ask the librarian for books like this one and you'll get good results, often ones you'd never have found by title. Ask for the exact page you edited this morning and you're out of luck, because the shelving was done last night. The librarian will still hand you the closest match with full confidence, but it will be last night's copy.

How it works in four steps (no maths)

• ▹1. Chunk: split each source (a doc, a ticket, a file) into pieces small enough to embed. That's usually a few hundred tokens, sometimes split by heading or by function.
• ▹2. Embed: run every chunk through an embedding model so each one becomes a vector. Store the vector together with the original text and metadata such as the source path and embed time.
• ▹3. Index: build an approximate nearest neighbour (ANN) index, for example HNSW, which many popular vector DBs use or offer. 'Approximate' means it gives up a little accuracy for a lot of speed, so you don't have to compare the query against every stored vector.
• ▹4. Query: embed the user's or agent's question with the same model, ask the index for the k nearest chunks, and paste those chunks into the LLM's context.

People tune three settings. You'll see these names in most vector DB tutorials and vendor dashboards:

• ▹Chunk size (and overlap): chunks that are too small lose their context. Chunks that are too big average their meaning into mush and use up context window.
• ▹Embedding model: this decides what counts as 'similar'. A general-purpose text model and a code-tuned model will put the same snippet in different places. If you switch models, you have to re-embed everything.
• ▹Similarity threshold / top-k: how close a result has to be to count, and how many results you take. Too loose and you pull in noise. Too tight and you get nothing back.

Where it shines

A vector database works best when the content is mostly prose and changes slowly compared with how often you query it. In that case you really are asking 'find things with similar meaning', and a slightly stale index costs you almost nothing.

• ▹Support docs and runbooks: 'how do I rotate the API key' should match a page titled 'Credential lifecycle', even though the two share no words.
• ▹Design docs and ADRs: 'why did we choose Postgres over Dynamo' should find the decision record even if it never uses the word 'choose'.
• ▹Past tickets and incidents: 'has anything like this timeout happened before' is a similarity question by nature.
• ▹Long-term agent memory: preferences, past decisions and lessons the agent wrote down weeks ago. Each one is written once and read many times.

The failure explained: agent write velocity vs. index freshness

Back to the deleted function. Embedding pipelines often run as batch jobs: on a schedule, on push, or from a queue that drains when it can. A human developer might change a few files an hour, so a lag of several minutes rarely matters. A coding agent can change dozens of files in the same window and then immediately ask for context about the code it just changed. At that point the index keeps serving a version of the repo that no longer exists, and the agent has no way to tell.

Why is there no error? Cosine similarity only measures how closely two vectors point the same way. It has no notion of 'outdated', 'deleted' or 'superseded'. An old chunk about invoice formatting still matches a question about invoice formatting very well. As far as the database is concerned, the query succeeded. That's why your logs look clean while the agent heads the wrong way.

Why code is different

Prose roughly means what its words say. Code means what its names and structure say. Take `getUser(id)` and `getUserById(id)`. Their vectors will probably land very close together. But one might return null and the other throw, or one might hit the cache and the other the database, so they're different contracts. In code a single character can change the meaning completely, and embeddings tend to blur exactly those small differences.

Most questions an agent asks about live code are exact and structural:

• ▹Where is this symbol defined, and does it still exist? Use grep / ripgrep, which matches exactly and is always current.
• ▹Who calls it, and what breaks if I change it? Use the call graph or references from a language server (LSP), or a code index such as ctags or tree-sitter.
• ▹What just changed? Use git diff and git log, which are the source of truth for recent edits.

# Exact, current, zero lag
rg -n "formatInvoiceLegacy\(" src/

# What changed in the last hour of agent work (needs reflog history; includes uncommitted edits)
git diff --stat HEAD@{1.hour.ago}

# Who references a symbol (via your LSP / editor tooling, or a quick approximation)
rg -n "\bgetUserById\b" --type ts

That's why many coding agents rely on agentic search over the file system. Anthropic has said publicly that Claude Code uses tools like grep, glob and file reads instead of a pre-built vector index of your repo. Cursor's agent combines its semantic codebase search with grep and direct file reads. The reasoning is the same in both cases: the file system is always current, and the agent can keep searching until it finds the exact thing.

What we changed: split retrieval plus a freshness check

We kept the vector database but gave it a narrower job. Live code is now retrieved with grep, the call graph and recent diffs. Docs, tickets and long-term memory still go through the vector DB. No chunk from the vector DB reaches the agent until it passes a freshness check.

import hashlib, os

def is_fresh(chunk) -> bool:
    """chunk.meta holds what we recorded at embed time."""
    path = chunk.meta["source_path"]
    if not os.path.exists(path):
        return False  # source deleted since embedding
    # Cheap check first: file untouched since we embedded it
    if os.path.getmtime(path) <= chunk.meta["embedded_at"]:  # epoch seconds
        return True
    # mtime moved; confirm with content hash
    with open(path, "rb") as f:
        return hashlib.sha256(f.read()).hexdigest() == chunk.meta["source_sha256"]

def retrieve(query, k=8):
    hits = vdb.search(query, k=k * 2)  # over-fetch, some will be dropped
    return [h for h in hits if is_fresh(h)][:k]

The check is short and cheap to run, and it fixes the silent part of the failure. A stale chunk is now either dropped or replaced by a direct read of the current file, so the agent never acts on it. Log every chunk you drop. A high drop rate means that content belongs on the grep side of the split, not the vector side.

Common pitfalls

• ▹Indexing the whole repo by default because the RAG tutorial did.
• ▹Treating similarity score as confidence. A high score means the text is on the same topic, not that it's correct or current.
• ▹Switching embedding models without re-embedding the whole corpus. Vectors from two different models can't be compared.
• ▹Chunking code by fixed token counts, which cuts functions in half. If you have to embed code, chunk it by symbol.
• ▹Leaving embed time and source hash out of the chunk metadata. Without them you can't run a freshness check later.

Try this week

Write down five 'where is X used' or 'what calls Y' questions about your own repo. Choose symbols that an agent or a teammate changed this week if you can. Ask your agent each question twice: once limited to embedding search, and once limited to ripgrep plus file reads. Mark each answer right, wrong, or confidently wrong. An answer is confidently wrong when it names a file, line or function that doesn't exist or is no longer used. Bring the confidently wrong count to your team.

Glossary card

A vector database finds what is similar, not what is true right now.

Tomorrow, Day 26: once retrieval gives you the right material, you have to decide how much of it goes into context, in what order, and what to leave out.

Extend your knowledge

• ▹Read the HNSW paper by Malkov and Yashunin, 'Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs'. It describes the index that many vector DBs use underneath.
• ▹Look at the pgvector extension for Postgres. If your docs and tickets already live in Postgres, it's often all you need, and you can skip running a separate vector DB.
• ▹Find how Anthropic's Claude Code team explains choosing agentic search (grep, glob, read) over a pre-built vector index, and compare it with how Cursor describes its codebase indexing.
• ▹Get properly comfortable with ripgrep and your language server's 'find references'. For live code, these are the retrieval tools that never go stale.