Context precision

Day 24: Context Precision — Why the Wrong File Hurts More Than the Missing One

The right file was in the window. Recall was 100%. The agent still wrote the feature on top of a deprecated helper. When a coding agent gets something wrong, most people assume retrieval missed a file. A lot of the time the right file is there and loses to a different file that looked more convincing. Today you'll learn to measure that other file, and to decide how much context to send based on whether the agent's output is correct, not on whether the right file appeared.

1. The incident: recall was perfect, the output was wrong

We ran into this while wiring retrieval into an internal coding agent. The task was to add a refresh-token check to a new endpoint. The retriever sent the top 30 files, and the correct, current auth module was one of them, so recall@30 = 100%. The agent built the feature on the deprecated helper anyway. Here are the two paths:

services/auth/token_validator.py          <- canonical, current, what we needed
services/auth/legacy/token_validator.py   <- deprecated, kept for one old client

# Both define:   def validate_token(token: str, *, audience: str) -> Claims
# Both have docstrings, both have tests, both import the same JWT library.
# Only one has a single line near the top:  # DEPRECATED: do not use for new code

Same file name, same function signature, same imports. The only differences are one directory segment and one comment line. An embedding model sees two almost identical vectors. The agent saw two equally good examples of "how we validate tokens here", and the legacy one happened to have a code path that looked closer to the task.

2. The uncomfortable question

If the right answer was in the window, what failed? Retrieval didn't, at least by the metric we were using. That metric asked one question: was the needed file there? It never asked what else was there. For a coding agent, the second question matters at least as much as the first. Usually it matters more.

3. Two numbers, in plain words

• ▹Recall@k: of the files the task actually needed, what share made it into the top k? In other words, did we include what was needed?
• ▹Precision@k flips it around: of the k files we sent, what share was actually needed? How much of what we sent was useful?

Both numbers come from the same retrieval run. Our dashboard showed only the first. With 100% recall and 7% precision, 93% of what the agent treated as evidence was noise, and some of that noise was dangerous.

4. Why low precision hurts agents more than search users

A person scanning search results skims, skips the obvious junk and clicks the one that looks right. An agent doesn't skim. It treats everything in its context as evidence of how this codebase works, and it acts on that evidence by writing code. Low precision breaks agents in three different ways:

• ▹Imitation. The agent copies a pattern from a file it should never learn from. In our case it used the legacy `validate_token` with a hard-coded audience, because that's what the deprecated file does.
• ▹Mis-targeting. The agent edits the wrong file. Ask it to fix a token-validation bug and it patches `auth/legacy/token_validator.py`. The tests pass, because legacy has its own tests, and the real bug ships.
• ▹Dilution is the quiet one. The right file is present but gets less attention. The canonical module sits at position 17 of 30, and the agent never reads the function it needed because other files had already answered the question "well enough". ("Lost in the Middle", Liu et al., 2023, shows the long-context version of this effect.)

5. Distractors are the real enemy

Some noise is harmless. If a CSS file gets pulled into a backend auth task, it wastes tokens, but the agent can see it doesn't apply. The damage comes from files that look relevant and are wrong. We call them distractors:

• ▹Near-duplicates: two implementations of the same thing living in different services.
• ▹Old versions such as `legacy/`, `v1/`, `_old` and `deprecated/` paths, plus whatever a migration left behind.
• ▹Test fixtures and mocks. `fixtures/fake_token_validator.py` looks like real code and misrepresents real behaviour.
• ▹Vendored copies in `vendor/`, `third_party/` or copied-in SDKs, whose patterns don't follow your conventions.
• ▹Generated code, like protobuf stubs, ORM output and snapshots. It looks authoritative, but nobody should edit it.

That calls for a third metric next to recall and precision. Distractor rate@k = the share of the k retrieved files that aren't needed but look like a needed file (same name or symbol, or a legacy, fixture, vendored or generated path). Precision tells you how much noise you sent. Distractor rate tells you how much of that noise can mislead the agent.

6. The 1M-token trap

Long context windows made "just include everything relevant" feel safe. A bigger window lets you send more. It doesn't make a deprecated look-alike any less convincing. We covered long-context degradation on earlier days, so today's point is narrow: window size is how much you can send, not how much you should. Anthropic's "Effective context engineering for AI agents" post and Chroma's "Context Rot" report reach the same conclusion. Performance drops as context grows, so treat context as a limited resource and don't fill the window just because it's there.

7. How to measure it this week

You don't need a research benchmark. You need 20–30 tasks from your own repo and honest labels.

• ▹Pick 20–30 recently merged PRs that a coding agent could plausibly have done, such as bug fixes, small features and refactors. Leave out the giant migrations.
• ▹Turn each PR into a task. The PR title plus the linked issue is the prompt, and the base commit is the repo state.
• ▹Label the needed files: everything the PR touched, plus first-hop imports the change depends on (types, the interface being implemented). Stop at one hop. Go further and everything counts as "needed".
• ▹Label known distractors per task. If the PR touched `auth/token_validator.py`, then `auth/legacy/token_validator.py` is a labelled distractor. Add path rules for legacy, fixtures, vendor and generated code on top.
• ▹Run your retriever on every task and log recall@k, precision@k and distractor hits.

import re
from pathlib import PurePosixPath

DISTRACTOR_PATHS = re.compile(r"(^|/)(legacy|deprecated|v1|vendor|third_party|fixtures|mocks|__snapshots__|generated)(/|$)")

def is_distractor(path: str, needed: set[str]) -> bool:
    if path in needed:
        return False
    same_name = PurePosixPath(path).name in {PurePosixPath(n).name for n in needed}
    return same_name or bool(DISTRACTOR_PATHS.search(path))

def score(retrieved: list[str], needed: set[str]) -> dict:
    k = len(retrieved)
    hits = [f for f in retrieved if f in needed]
    distractors = [f for f in retrieved if is_distractor(f, needed)]
    return {
        "k": k,
        "recall": len(set(hits)) / len(needed),
        "precision": len(hits) / k,
        "distractor_rate": len(distractors) / k,
        "distractors": distractors,  # log names, not just counts: you'll want to read them
    }

Keep a table with one row per task. You'll learn the most from the distractor-names column.

| task_id | PR    | k  | needed | recall@k | precision@k | distractor_rate | distractor files                 | agent pass? |
|---------|-------|----|--------|----------|-------------|-----------------|----------------------------------|-------------|
| T01     | #1423 | 30 | 2      | 1.00     | 0.07        | 0.10            | auth/legacy/token_validator.py … | no          |
| T02     | #1431 | 30 | 3      | 0.67     | 0.07        | 0.03            | tests/fixtures/fake_billing.py   | yes         |
| ...     |       |    |        |          |             |                 |                                  |             |

"Agent pass?" means the agent's patch passes the PR's own tests, or tests you wrote for it. If you log only retrieval metrics, you're measuring the window again and not the outcome.

8. Sweep k against task success, not against recall

Recall can only go up or stay flat as k grows, so a recall-vs-k chart will always tell you to send more. Size k from the agent's pass rate:

• ▹Run the agent on every eval task at k = 5, 10, 20 and 40.
• ▹Do at least 3 runs per configuration. Agents are nondeterministic, and with one run per task you'll mistake noise for signal.
• ▹Plot pass rate against k, and put recall@k and distractor rate@k on the same chart.

The curve usually looks like this. Pass rate climbs while the needed files start making it in, flattens once recall is mostly covered, then falls as k keeps growing and distractors get in. Use the k at the peak. In our runs the peak came well before recall reached its maximum. If your curve never drops, your index is probably clean already, and that's useful to know too.

9. Curation levers, roughly cheapest first

• ▹Exclude at index time. If a file isn't indexed, it can't be retrieved as a distractor. Put `legacy/`, `vendor/`, `third_party/`, generated stubs and snapshots in an ignore file. It costs almost nothing, and in our experience it gave the biggest improvement.
• ▹Metadata filters. Tag each chunk with its owner (from CODEOWNERS), last-modified date and a deprecation flag parsed from `@deprecated` or `# DEPRECATED` markers. Then filter or down-weight at query time.
• ▹Dedupe near-duplicates. Cluster chunks by embedding similarity or MinHash and keep one per cluster, preferring the most recently changed, non-legacy path.
• ▹Canonical-file boost in reranking. Give the reranker from Day 20-ish a new feature: is this file imported by many others, recently edited, owned by an active team, outside legacy paths? When canonical code and a look-alike tie, canonical should win.
• ▹Pull instead of push. Preload a small, high-precision set and give the agent tools (`search_code`, `read_file`, `find_references`) to fetch more when it actually needs it. That's tomorrow's topic.

10. Our after state, and the honest trade-off

We made three changes: k went from 30 to 8, legacy and vendored paths came out of the index, and the reranker got a canonical-file boost. Recall dropped a little, because some tasks now miss a needed file in the top 8. Task success went up, and the deprecated-helper failure stopped showing up in our eval set.

There is a cost. The agent sometimes has to fetch a file it wasn't given, which adds a tool call and some latency. We accept that. A fetch shows up in the trace and is cheap to fix. A confident edit to the wrong module can pass review looking like a good change.

11. The rule of thumb

A missing file makes the agent ask for it. A wrong file makes the agent confidently do the wrong thing. Size your context for the second case.

• ▹High k is fine when the agent can't fetch more context (one-shot generation, no tools) and the index is already clean.
• ▹Avoid high k when your repo has legacy copies, forks or vendored code. In my experience that's most repos older than a few years.
• ▹Common pitfall: the team dashboard shows only recall@k. Put precision@k, distractor rate and pass rate next to it, or the dashboard will keep pushing for more context.
• ▹Another pitfall is labelling "needed" too generously, for example whole import trees. Precision will look fine and tell you nothing.

12. Bridge to Day 25

Once you can measure precision, the next question is who decides what goes into context. It could be a fixed retriever picking top-k up front, or the agent itself searching and reading through tool calls as it works. Several coding agents, Claude Code among them, rely heavily on the second approach. Tomorrow covers agentic retrieval and how to evaluate it with the three metrics you set up today.

Extend your knowledge

• ▹This week, build a 20–30 task eval set from your own merged PRs and run the k = 5/10/20/40 sweep, with at least 3 runs per setting.
• ▹Read Anthropic's engineering post "Effective context engineering for AI agents" and Chroma's "Context Rot" technical report for the wider case for treating context as a limited resource.
• ▹"Lost in the Middle: How Language Models Use Long Contexts" (Liu et al., 2023) is the paper to read on dilution and position effects.
• ▹Look at the context precision and context recall metrics in the RAGAS library. RAGAS context precision is rank-weighted and LLM-judged by default (non-LLM variants exist too). Compare it with the file-level labels you built today.