0.89 Similarity, Wrong Function: How an Agent Wired a Rate Limiter Into an Auth Route

Day 12: Topically Similar Isn't Behaviorally Equivalent

Your vector search comes back with 0.89 cosine similarity and the agent treats that number like a verdict. It isn't one. Embeddings are the default retrieval layer in nearly every agent stack you'll build this year — RAG, tool selection, code search, all of it — and they're genuinely good at finding things that are about the same subject. What they cannot tell you is whether two things do the same thing. Let an agent trust that score without a second check, and the gap between 'similar' and 'equivalent' ships as a bug wearing a confidence badge.

The incident

Picture an agent running a tool-selection loop, about to wire a new route into the app. Before it does, it needs to find the auth middleware, so it queries the vector store with 'auth middleware.' Top hit: rateLimiter.js, at 0.89 similarity. authMiddleware.js sits right behind it at 0.83 — close enough that a human skimming the list would probably open both. The agent doesn't skim. It reads 0.89 as 'found it' rather than 'best guess,' and attaches the new route to the rate limiter. The route ships looking protected. It isn't. Anyone with a valid IP and a little patience walks straight through — there's no identity check anywhere in the chain.

Why the vectors agreed

Both functions do the same physical job: intercept a request, check something, then either call next() or short-circuit with an error. Both are built from the same tiny vocabulary — req, res, next, status, return — that shows up in basically every piece of Express middleware ever written. An embedding model mostly reads code as token distribution plus structural shape. It isn't tracing what the code enforces at runtime; it's pattern-matching on what the code looks like on the page. Shape and vocabulary dominate the vector. The actual guard condition — the part that decides whether this function protects anything at all — barely moves the needle.

function rateLimiter(req, res, next) {
  const count = hits.get(req.ip) || 0;
  if (count > LIMIT) {
    return res.status(429).send('Too many requests');
  }
  hits.set(req.ip, count + 1);
  next();
}

function authMiddleware(req, res, next) {
  const token = req.headers.authorization;
  if (!isValid(token)) {
    return res.status(401).send('Unauthorized');
  }
  next();
}

Same skeleton, every time: intercept, check a condition, short-circuit or call next(). The embedding model has no slot for 'this checks who you are' versus 'this checks how often you've asked.' It sees two near-identical shapes and calls that similarity.

Name the pattern: topical similarity vs. behavioral equivalence

Topical similarity is what embeddings actually measure — these two things are 'about' the same subject, use the same words, sit in the same neighborhood of the vector space. Behavioral equivalence is what you need for code — these two things produce the same result when you run them. In document search that gap rarely bites: it's unusual for two paragraphs to read almost identically and assert opposite claims, because prose gets written to be read literally. Code is a different animal. A small set of idioms — guard clause, middleware signature, try/catch, early return — gets reused across completely unrelated behaviors. Auth checks, rate limits, feature flags, request logging: all of them can compile down to 'if (condition) { short-circuit } next().' Same shape, and the consequences of swapping one for another are not remotely the same.

What actually worked: a secondary signal the embedding can't fake

• ▹Call-graph position — check what actually imports and wires the function into the route table, not what's sitting nearby in vector space
• ▹The guard condition itself — open the function and look at what it's actually checking: a token, a request count, a feature flag. That's the part the embedding glosses over
• ▹Static analysis as a narrowing pass — let the embedding produce a shortlist, then use something that reads real logic to pick a winner
• ▹A quick test before wiring anything in — call the candidate with a bad token. Does it actually reject the request, or just look like it would?

None of this is exotic. It's what any careful engineer already does before trusting a grep result — the fix here is just refusing to let the agent skip that step because the retrieval step handed back a number that looked confident.

The rule for Day 12

Treat a semantic search hit as a hypothesis, not an answer — especially the moment an agent is consuming it unsupervised and acting on it before any human glances at the diff. A similarity score tells you where something sits in vector space. It tells you nothing about what happens when the code runs.

Bridge to Day 13: if a similarity score can't be trusted blindly, what does an agent actually need to check before it acts on retrieved context?

Extend your knowledge

• ▹Look at how code embedding models (OpenAI's text-embedding-3, CodeBERT-style models) are actually trained — mostly token co-occurrence and code/docstring pairs, never execution traces. That training data is exactly why they capture shape over behavior
• ▹Look at how Sourcegraph Cody pairs embedding retrieval with keyword search and code-intelligence data — symbols, references, call hierarchy — instead of leaning on embeddings alone. Most serious code-search products treat vector similarity as one signal among several, not the final word
• ▹Try it yourself: embed two functions with identical control-flow shape but opposite behavior — auth check, rate limit, cache check — and compare the similarity scores directly. The numbers will be closer than you'd like
• ▹Read into 'RAG evaluation' practices. Most teams shipping agentic retrieval add a re-ranking or verification pass for exactly this reason: raw similarity isn't reliable enough to act on alone