Frequency bias in coding agents

Day 23: Frequency Bias, or Why Your Codebase Is the Prompt

Your coding agent pays more attention to your codebase than to your prompt. Once it's working in your repo, the files it reads shape its output more than anything you type. If the most common pattern in those files is the wrong one, the agent will copy it, and every PR it opens adds one more copy.

The hook: 40 versus 3

At PhoenixDX we had two auth helpers in the same service. The old one was deprecated. A comment said so, and an ADR explained why we'd moved off it. The new one was correct, tested and the recommended choice. The old helper had about 40 call sites. The new one had 3.

We asked a coding agent to add a few new endpoints, and it used the deprecated helper every time. It didn't hide this. The PR description said, more or less, that it went with the old helper to stay "consistent with existing patterns". Given what it had seen, that was a sensible call. Nothing was wrong with the agent. It did exactly what the codebase taught it to do.

The agent counted correctly. The problem was what it was counting.

Reframing 'bias' for this course

When most people hear 'AI bias', they think of fairness and demographics. Those problems are real. Today, though, we're using a narrower engineering definition:

• ▹Bias = a systematic tilt in a model's output that comes from what the model was shown.
• ▹"Systematic" means it leans the same way every time. It isn't random noise.
• ▹"What it was shown" includes training data. For coding agents, it mostly means what's in the context window right now: your files, your examples, your conventions.

This builds on the days about context windows and in-context learning. The model learns from the examples in its context. Today's point is that most of those examples come from your repo.

The mechanism in one picture

When you give an agent a task, it searches your repo (grep, file reads, semantic search, open tabs) and pulls the relevant files into context. Those files become its few-shot examples. In-context learning treats repetition as signal, so if 40 examples do X and 3 do Y, X reads as 'how this codebase does things'. The model has no reliable way to tell 'common' from 'correct' unless something in context tells it. A single deprecation comment is easy to outvote, so frequency ends up standing in for correctness.

Here's an analogy. A new hire never reads the ADRs. They pick up the house style by opening a dozen files and copying whatever appears most often. They're careful and consistent, and they're wrong every time the most common pattern is a legacy one. A coding agent is that new hire, except it works much faster and never asks anyone on Slack whether the pattern is still current.

Why it compounds: the feedback loop

Before agents, legacy patterns spread only as fast as people copy-pasted them, and code review caught some of that. Agents speed the loop up:

• ▹Generation 0: 40 old usages, 3 new.
• ▹Generation 1: the agent writes 10 new files and copies the old pattern into each. Now it's 50 to 3.
• ▹Generation 2: the next agent run, or a teammate's agent, retrieves from a repo where the old pattern is even more dominant, so the tilt gets stronger.
• ▹The new pattern's share keeps falling even though nobody chose the old one. Every generated file is one more vote for it.

Conceptually, the debt curve looks like this. Before agents, legacy usage grows slowly, roughly in step with team size. With agents, it grows in step with agent output, which is far higher. I want to be precise about what this claim is. It's a feedback loop I've watched happen, and it follows from how in-context learning works. I haven't measured it as a law with a fixed rate. How fast it compounds depends on how much code your agents write, how retrieval chooses files and how closely reviewers look. The direction is predictable, though. Without some correction from outside the loop, the majority pattern wins and keeps growing.

Spot it in your own repo in 15 minutes

List the places where your team has an 'old way' and a 'new way'. Typical candidates are auth helpers, HTTP clients, logging, date libraries, ORM query styles, error handling and test utilities. Count the occurrences for each pair.

# Illustrative names: swap in your own competing patterns
# Auth helper: old vs new
rg -c "getUserFromToken\(" --type ts | awk -F: '{s+=$2} END {print "old auth:", s}'
rg -c "requireSession\("    --type ts | awk -F: '{s+=$2} END {print "new auth:", s}'

# HTTP client: old vs new
rg -l "from 'axios'" src | wc -l
rg -l "from '@/lib/http'" src | wc -l

# Dates
rg -l "moment\(" src | wc -l
rg -l "date-fns" src | wc -l

Rule of thumb: unless something tells the agent otherwise, expect it to pick the bigger number. Put the pairs in a small table with three columns: pattern, old count, new count. In every row where the old count is higher, the agent is already biased the wrong way.

• ▹Pitfall: counting only in src/. Agents also read tests, scripts and examples, so count those too.
• ▹Pitfall: skipping generated or vendored code. If the agent can retrieve it, it gets a vote.
• ▹Pitfall: trusting the deprecation comment. One /** @deprecated */ tag is easily outvoted by 40 call sites.

The fixes, ordered by leverage

1. Delete or quarantine deprecated patterns before you roll out agents. This has the most leverage because it takes the votes away. Either migrate the call sites, or move legacy code somewhere agents are told to ignore (a legacy/ directory plus an ignore rule). With the old examples gone, the agent has nothing old to copy. Agents can do a lot of this migration themselves now, as long as you point them explicitly at the new pattern.

2. Keep explicit golden-path files and point your agent instruction file at them. Pick one canonical example for each concern and name it in CLAUDE.md, AGENTS.md, .cursor/rules or your Copilot instructions. The agent then reads a known-good example first, before retrieval surfaces whatever happens to be most common.

# CLAUDE.md (excerpt)

## Golden paths: read these before writing similar code
- Auth in route handlers: `src/api/orders/create.ts` (uses `requireSession`)
- Outbound HTTP: `src/lib/http/example.ts`

## Do not use
- `getUserFromToken`: deprecated, see docs/adr/0012-session-auth.md
- `axios` directly: use `@/lib/http`

If you see the deprecated pattern nearby, do NOT copy it for consistency.
The golden path wins over local consistency.

3. Add a lint or CI rule that fails on the deprecated pattern. That way the build corrects the agent instead of you. Agents working in a loop run the linter and tests, read the error and fix the code. A failing rule is a correction that's always there and never gets tired.

// eslint.config.js (excerpt)
export default [
  {
    rules: {
      'no-restricted-imports': ['error', {
        paths: [{
          name: '@/auth/legacy',
          importNames: ['getUserFromToken'],
          message: 'Deprecated. Use requireSession from @/auth (ADR-0012).'
        }, {
          name: 'axios',
          message: 'Use @/lib/http instead.'
        }]
      }]
    }
  }
];

4. Write the ADR as something the agent has to read first, not just as a record of history. A traditional ADR explains why a decision was made, for people reading it months later. An agent-facing ADR also states the rule in one line, names the allowed and forbidden APIs and links to the golden-path file. Reference it from your instruction file so the agent actually loads it.

• ▹When NOT to bother: one-off scripts and throwaway prototypes that no agent will touch again.
• ▹When it matters most: right before agent use spreads from one or two enthusiasts to the whole team. The compounding is driven by agent volume.
• ▹Pitfall: doing only fix #2. Instructions help, but 40 counterexamples in context still pull hard. What makes the fix stick is removing the votes (fix #1) and enforcing the rule (fix #3).

What happened after we did it

We migrated the deprecated auth call sites, added a lint rule against the old helper and pointed our agent instructions at a golden-path handler. Deprecated usages dropped to zero. The agent switched to the new helper by default, and we never changed the task prompt. Its PR descriptions still said 'consistent with existing patterns', and now that was exactly what we wanted.

The lesson: the cleanup was the prompt change. We used to file refactoring under housekeeping, something to get to eventually. With agents in the repo, removing a deprecated pattern changes what the model is conditioned on, which makes it prompt engineering.

Course bridge

Today was about bias in what the model reads. Day 24 covers bias in who checks the work. If the reviewer is the same model, or a model trained on similar data, it tends to have the same blind spots and can approve the same mistakes. That leads into the multi-agent days.

Takeaway

Agents don't follow your best practices. They follow your most common ones.

Extend your knowledge

• ▹This week, run the grep audit on your main repo and build the old-vs-new table. Choose the worst pair and migrate it before your next agent rollout.
• ▹Read Anthropic's Claude Code docs on CLAUDE.md and memory files, plus the AGENTS.md convention. Then add 'Golden paths' and 'Do not use' sections to your own file.
• ▹Look at ESLint's no-restricted-imports and no-restricted-syntax rules, or Semgrep for patterns across languages, to turn deprecations into build failures.
• ▹Go back to Michael Nygard's original 'Documenting Architecture Decisions' post and rewrite one of your ADRs so it opens with a one-line rule an agent can follow.