Exemplar audits

Day 21: Your codebase is the style guide your agent actually reads

Your agent read the style guide. Then it read forty files that break it, and it trusted the files. If your agent keeps giving you code that's "almost right but off-style", adding more rules to the prompt usually won't fix it. The agent copies your code, so change the code it copies.

1. The PR review

I was reviewing an HTTP handler that an agent had written. It ran and the tests passed. But every operation was wrapped in the same try/catch-and-log: catch the error, log it, return a generic 500, move on. Our style guide had banned that pattern more than a year earlier. We wanted typed errors that propagate to a single error middleware, so stack traces survive and status codes stay correct. The ban was in CLAUDE.md, in plain English.

// What the agent wrote (banned by our guide)
export async function getInvoice(req: Request, res: Response) {
  try {
    const invoice = await invoiceService.find(req.params.id);
    res.json(invoice);
  } catch (err) {
    logger.error(err);
    res.status(500).json({ message: "Something went wrong" });
  }
}

// What CLAUDE.md asked for
export async function getInvoice(req: Request, res: Response) {
  const invoice = await invoiceService.find(req.params.id); // throws NotFoundError, etc.
  res.json(invoice);
} // errors handled once, in errorMiddleware

My first instinct was that the model had ignored the instructions and I needed a stronger prompt, maybe one in capitals. That was the wrong diagnosis. If you keep one thing from this lesson, keep that.

2. The trace: where did that pattern come from?

So we left the prompt alone and debugged the output instead. We pulled the session transcript, checked which files the agent had actually opened during the task, and diffed its output against those files. (You could ask the agent afterwards, but its self-report can be incomplete. The transcript is more reliable.)

• ▹The try/catch-and-log shape matched three legacy handlers almost line for line. It had the same variable names and the same "Something went wrong" string.
• ▹Those three files sat in the same folder as the new handler. They were its closest neighbours, so they were the first examples the agent opened.
• ▹According to the guide, those files shouldn't have existed anymore. We'd meant to migrate them and never did.
• ▹Across the repo, the banned pattern still showed up in dozens of places. The rule against it showed up in exactly one.

Roughly speaking, the docs had one vote and the code had dozens. The agent sided with the majority.

3. The mechanism in plain words

Before writing anything, an agent like Claude Code, Cursor, or Copilot gathers context. It opens files near the one it's editing, searches for similar functions, and follows imports. Every file it opens shows it how things are done in this repo. A language model is built to continue patterns, so if its context is full of one shape, that shape is very likely what it writes.

Your rule in CLAUDE.md is one sentence. The examples the agent reads add up to hundreds of lines. They're concrete, they sit right next to the edit, and they show the exact syntax to produce. The rule describes what you want, but the examples show it in full detail. When the two disagree, the examples usually win.

Sidebar for newcomers: why "just tell it" isn't enough

Picture onboarding a new developer. You hand them a style guide that says "never do X". Then you show them twenty files that all do X and ask them to add a twenty-first that "fits in". Most people would copy the twenty files and assume the guide was out of date. An AI agent behaves much the same way, only faster and more consistently. It isn't ignoring the rule. It weighs the rule against the evidence and concludes that the evidence is what you actually want.

4. The reframe: drift is a mirror, not a malfunction

Once you see the mechanism, convention drift in agent code stops being an annoyance and becomes useful information. Agent output is a sample of how your codebase is really written. When the agent strays from the guide, it's usually because your team strayed first, often without anyone noticing.

• ▹The agent keeps writing pattern X despite a rule → X is still common in the files it reads.
• ▹The agent follows the rule in one module and breaks it in another → the second module has its own local conventions.
• ▹The agent invents a pattern that appears in neither the docs nor the code → this one is a real model error. In my experience, that's much rarer than people assume.

In practice, this means that before you blame the model, you should check whether it's just reflecting your own code back at you.

5. How this connects to earlier days

Earlier in the course we covered how to write good context: CLAUDE.md, rules files, and task framing. All of that still matters. Today adds one point. Your code is the largest context file you have, and you can't opt out of it. Every agent that edits the repo reads it, whether you curated it or not. So curating context also means curating the code the agent is likely to read.

6. The tactic: an exemplar audit in four steps

An exemplar is a file the agent copies from. Usually a handful of files do most of the teaching. The audit finds those files so you can fix them first.

• ▹Step 1: Collect evidence. Pull recent agent-written diffs that got style comments in review. Ten to twenty is enough to start. Tag each one with the convention it broke.
• ▹Step 2: Find the source. For each diff, work out which existing files it resembles. The best signal is the agent's own file-read history. If you don't have that, a simple similarity grep for the distinctive shape of the off-style code will do.
• ▹Step 3: Rank. Count how often each file shows up as a likely source. For us, a handful of files accounted for most of the copying.
• ▹Step 4: Fix or fence. Refactor those files to the current convention. If you can't refactor them yet, fence them off with a header comment telling readers, human or agent, not to imitate them. Only after that should you touch prompts or rules.

Step 2a: use the agent's read history. Claude Code stores session transcripts as JSONL under ~/.claude/projects/, so you can count which files the Read tool opened. The transcript format can change between versions, so check yours before relying on this:

# Count which files the agent read most across recent sessions
jq -r 'select(.type=="assistant")
       | .message.content[]?
       | select(.type=="tool_use" and .name=="Read")
       | .input.file_path' \
   ~/.claude/projects/<your-project>/*.jsonl \
 | sort | uniq -c | sort -rn | head -20

Step 2b: run a similarity grep for the banned pattern. It's crude. It assumes the log call sits within two lines of the catch and that file names contain no hyphens. Still, it shows you where the pattern is concentrated:

# Files containing the catch-and-log shape, ranked by occurrences
grep -rn --include='*.ts' -A2 'catch (err)' src/ \
 | grep 'logger.error(err)' \
 | cut -d- -f1 | sort | uniq -c | sort -rn | head -10

Step 4: write the fence. Keep it short and specific, and point to the file the agent should copy instead:

/**
 * LEGACY — DO NOT IMITATE.
 * This handler uses try/catch-and-log, which is banned (see CLAUDE.md#errors).
 * For new handlers, copy src/handlers/orders/getOrder.ts instead.
 * Migration ticket: <your tracker id>
 */

That pointer matters. A fence that only says "don't" leaves the agent with nothing to follow. A fence that says "copy this file instead" gives it a better example.

7. What changed for us

• ▹We refactored a small set of the top-ranked exemplar files and fenced a few more that we couldn't touch yet. Style comments on agent PRs dropped noticeably, and reviewers could focus on logic instead of patterns and formatting.
• ▹We changed almost nothing in the prompts, and that was the point.
• ▹The audit also had an uncomfortable side effect. It surfaced two "conventions" in the guide that the team had quietly abandoned. Nearly all recent human-written code broke them as well, and nobody had complained.
• ▹For those two, we deleted the rules from the guide instead of fixing the code. The guide had been describing a team we no longer were.

That second outcome is a real benefit of the audit. A style guide that contradicts the code is worse than having no guide, because it confuses humans and agents alike. The audit forces you to decide: either the rule is real and you fix the code, or the rule is dead and you delete it.

8. Limits and honesty

• ▹This doesn't replace deterministic checks. Even with clean exemplars, copying still produces mistakes, and probabilistic behaviour can always drift. Lint rules, type checks, and CI gates catch what copying gets wrong. The next lesson covers enforcement that can't be talked out of.
• ▹Small or very consistent repos see less of this effect. If your code already matches your guide, there's no counter-example for the agent to copy.
• ▹An exemplar audit isn't a one-off job. Legacy code keeps accumulating, and code written by agents becomes the next exemplar. Merge one off-style agent PR and it becomes part of the sample for every future run.
• ▹Fences are a stopgap. A file marked "legacy, do not imitate" for two years has become one of your conventions too. Keep track of your fenced files and migrate them.

When to use this, and when not

• ▹Use it when the same style violation keeps coming back in agent PRs even though the rule is clearly documented.
• ▹Use it right after you change a convention, while the old version still dominates the repo.
• ▹Skip it when the violation is random and one-off. That's more likely a model or task-framing error, so fix the context for that task.
• ▹Skip it when the rule can be enforced mechanically and cheaply. Write the lint rule first, because it's faster and stricter.

Common pitfalls

• ▹Adding more rule text, or more capital letters, to CLAUDE.md instead of tracing the source. That's just one more sentence against dozens of examples.
• ▹Fencing files without pointing to a correct replacement exemplar.
• ▹Auditing the whole repo at once. Start from flagged diffs, because they tell you which files matter.
• ▹Assuming the guide is right. Sometimes the code is right and the guide is out of date.

9. Exercise: predict, then check

• ▹Pick one convention you care about, such as error handling, how you name async functions, or how you structure tests.
• ▹Count it. Use grep to count the files that follow it and the files that break it, especially in the folders where agents do most of their work.
• ▹Write down a prediction: if I ask the agent to add a new file here, which version will it write?
• ▹Run the task with your normal agent setup, without any special prompt.
• ▹Compare. If your prediction was right, you've just measured the convention your team really follows. Then decide whether to fix the exemplars or delete the rule.

Extend your knowledge

• ▹This week, run the exercise above on your own repo for one convention. Write down what you predicted and what the agent actually wrote.
• ▹Read Anthropic's Claude Code documentation on memory and CLAUDE.md. Then compare what it says the agent loads with what your session transcripts show it actually read.
• ▹Go back to "in-context learning" in the GPT-3 paper (Brown et al., 2020, "Language Models are Few-Shot Learners"). Your codebase is effectively one huge few-shot prompt.
• ▹Look at your linter's options for banning patterns, such as ESLint's built-in no-restricted-syntax rule or a custom rule plugin. You'll need them for the next lesson on deterministic enforcement.