The RAG Bug That Never Throws an Error

Day 10 — Embeddings Are a Snapshot, Not a Live Index

Run the embedder once, dump the vectors into a store, ship the RAG feature, move on. That's the mental model most teams carry into production, and it holds up right up until it doesn't. Six weeks later the coding agent is confidently retrieving the wrong context, and nobody can say why — because this kind of failure doesn't look like a failure.

That's exactly what bit us at PhoenixDX. Our coding agent started pulling a stale auth-flow explanation to justify changes in a module we'd refactored a month earlier. Nothing about the retrieved code screamed "wrong" — it was adjacent, plausible, and out of date, which is worse. The agent's own reasoning gave the PR a green check, a reviewer skimmed the citation and trusted it, and everyone moved on. We only found out two days later, tracing a broken downstream service back to a vector store nobody had thought about since the day we turned it on.

Quick refresher: what an embedding actually captures

Quick refresher — from Day 1-9 you already know an embedding model turns "similar meaning" into "nearby points in vector space." An embedding is a fixed-size vector that captures the meaning of a chunk of text, or code, at the exact moment it was generated. The word doing all the work there is moment. The model isn't watching your codebase. It takes one snapshot of one chunk, freezes that meaning into a vector, and hands it to your vector store. Nothing about it updates on its own — it sits there until something tells the pipeline to regenerate it, and that trigger is almost always "this file changed on disk," a far cruder signal than "this code means something different now."

The actual bug: two things drift independently

Most embedding pipelines re-chunk and re-embed on file save. That conflates two things that don't move together: chunk boundaries — where you slice the file up for embedding — and semantic content — what the code actually does. Refactors break that pairing, and they break it in two opposite directions:

• ▹Meaning moves without the file moving: you inline a function, flip a default, or push business logic from one module into a caller. The file that got saved isn't the file whose meaning changed, so it never gets re-embedded.
• ▹The file moves without meaning moving: a rename, a lint pass, a formatting sweep, a reordered import block — all trigger a full re-embed of code whose semantics haven't shifted at all, which just burns compute on a no-op.
• ▹Chunk boundaries silently shift: add ten lines to a function and your chunker — fixed-size or AST-based — slices the file differently now. Last week's chunk and this week's "same" chunk aren't even comparable, but the index treats them as one continuously-updated thing.

In our incident, the refactor pulled a validation rule out of the auth module and into shared middleware — but the auth file's diff was tiny, just a changed function signature. The file-save trigger fired, dutifully re-embedded a chunk that had barely changed, and never touched the middleware file where the actual meaning now lived. The index kept sending agents back to an empty shell.

Why nobody noticed: retrieval fails plausibly, not loudly

A stale embedding doesn't throw an error. Cosine similarity still returns a score, the retrieved chunk still contains real words about auth and validation, and the LLM — trained to be helpful with whatever context it's handed — writes a fluent, confident answer on top of it. It's a close cousin of hallucination, except the source material is real code that used to be true. Both the agent and the human reviewer pattern-match on "this looks like the right file," not "is this still the current truth" — because nothing in the UI distinguishes a fresh embedding from one that's three months stale.

How we actually traced it

The fix started with a question we didn't have tooling to answer: when was this chunk's meaning last true, versus when was its vector last computed? So we built it — a script that diffs each chunk's embedding timestamp against the git blame date of the last commit to touch the semantic region it covers (not just the physical lines in the chunk, but the functions and symbols it references, via a simple call-graph pass). Any chunk where the semantic region had a newer commit than the embedding got flagged as suspect. That one diff surfaced dozens of stale chunks across the codebase — not just the one that had already bitten us.

The fix that isn't 'just re-embed everything'

Re-embedding everything on every commit is the tempting fix, and it's the wrong one — expensive at scale, and it never touches the root cause, which is that mtime is the wrong invalidation signal. Treat your embedding index like any other cache: it needs an explicit invalidation policy, not a hope that the write trigger happens to line up with the thing you actually care about.

• ▹Invalidate on semantic diff, not file diff — track which chunks reference a changed symbol, function, or call path, not just which files got saved.
• ▹Version chunks instead of overwriting them — keep the old vector alive until the new one is confirmed working, so a bad re-embed can't silently swap working retrieval for broken retrieval.
• ▹Surface a staleness score on retrieval results — "embedded 47 days ago, code touched 3 days ago" lets the agent, and the reviewer, discount low-confidence context instead of trusting it blindly.
• ▹Re-embed on refactor-shaped commits specifically — big deletions, function moves, and import restructuring are cheap signals that meaning shifted, even when the file list is short.

The takeaway for Day 10

Embedding isn't a pipeline step you run once and forget — it's a cache with a decay curve, and every day since your last re-index is a day your RAG system's confidence and its accuracy quietly pull apart. If you're running agentic coding tools against a vector store in production, you need a staleness budget the same way you need a cache TTL: a real answer to "how old can this context be before we stop trusting it," not just "did the file get saved."

Tomorrow: the other side of retrieval. Once the agent has context — fresh or stale — how does it decide how much of it to actually use?

Extend your knowledge

• ▹Read up on cache invalidation strategies — TTL, event-driven, versioned — and map each one onto your own embedding pipeline. Which one are you actually running today?
• ▹Audit one production RAG index yourself: pull five retrieved results from real agent sessions and check embedding timestamp against last semantic change on the underlying code.
• ▹Look into AST-aware or call-graph-aware chunkers instead of fixed-size chunking — they make 'semantic diff' cheaper to compute because chunk boundaries track code structure instead of character counts.
• ▹If you're on a managed vector store — Pinecone, Weaviate, pgvector — check what invalidation hooks it actually exposes beyond 'upsert on write.' Most leave you to build the semantic-diff trigger yourself.