The PR was clean. Lint, tests and type checks all passed, and the diff was tidy. It also rebuilt an `InvoiceRepository` class in `packages/api/src/services/billing/`, which is the pattern we had just spent two sprints tearing out.
The agent wasn't hallucinating. It did exactly what line 14 of our CLAUDE.md told it to: "Data access goes through a repository class in `services/<domain>/`, e.g. `packages/api/src/services/orders/OrderRepository.ts`." Then it went looking for evidence, and there was plenty. Most of `services/` still used repository classes, because the move to plain query modules in `packages/api/src/data/` was only half done. So the rule was out of date, the codebase backed the rule up, and every check we had passed. That was the bug.
The surprise came later, when we audited the rest of the file. The obviously broken lines (commands that no longer existed, paths to modules we'd renamed) had been failing loudly for months and had cost us almost nothing. The lines that hurt were the ones that still worked. A CI check that only confirms commands run and paths exist would have waved line 14 straight through. `OrderRepository.ts` existed, and it existed precisely because the migration wasn't finished.
This is how we run it now. A CLAUDE.md is a set of claims about your codebase. Every claim either proves itself in CI or has a named owner. A line that does neither doesn't merge.
The audit: loud rot is cheap, silent rot multiplies
We printed the file, one line per row, and four of us went down it with one question: is this line true about the repo today? We weren't asking whether it was good advice, only whether it was true. For every line that wasn't, we asked a second question: what has it actually cost us?
The wrong lines sorted into three piles, and the costs were nothing alike:
- **Dead commands.** `pnpm test:api`, for instance, had been replaced by a filtered workspace command. The agent runs it, gets a non-zero exit, reads the error, finds the right command in `package.json` and moves on. The cost is a few wasted tool calls per session. Nobody noticed, because nothing broke.
- **Renamed modules and moved paths.** The file pointed at an auth middleware that a refactor had moved. The agent opens the path and gets file-not-found. It searches with ripgrep, finds the new location and moves on. That costs some tokens and some latency, and does no damage.
- **Stale conventions.** These were line 14 and a couple of siblings: a logging helper we were phasing out, and an error-wrapping style that half the codebase had already dropped. Nothing fails here. The agent follows the rule, finds legacy code that agrees with it, and writes more legacy code. Every one of those PRs pushes the old pattern's share up, so the next agent finds even more evidence for the stale rule. Review caught some of these. Others we only found when a later migration PR came out bigger than it should have.
Stale conventions were only a handful of lines, but they did all the damage. Loud rot heals itself, because the agent hits a wall and finds a way around it. Silent rot compounds, because agents copy whatever pattern is most common.
Step 1: give every line a type
Every line in the file makes a claim, and different claims need different proof. We use four types:
- **Executable.** The line is a command, and you prove it by running it. Ours: `pnpm --filter api test`.
- **Referential.** The line names a path, module or symbol. You prove it by checking the path exists or resolving the symbol. Ours: "Feature flags are read through `isFlagEnabled` in `packages/api/src/config/flags.ts`."
- **Normative.** The line states a convention. To prove it, link it to the lint rule or canonical example that enforces it, then measure how much of the code actually follows it. Ours is line 14, after we rewrote it.
- **Judgment.** This is taste that no linter can check. Since it can't be proven, it gets an owner and an expiry date instead. Ours: "Prefer boring code in `billing/`. No clever generics, no metaprogramming."
Each claim gets an HTML comment on the line above it. Markdown renderers hide the comment and scripts can parse it. Whether the agent sees it depends on the tool, because some tools strip HTML comments before loading the file. So the prose has to make sense without the comment:
## Data access
<!-- rule: SVC-003 type=normative enforce=eslint:no-repo-class example=packages/api/src/data/orders/queries.ts adherence-min=0.3 adherence-note="migration in progress; raise the floor as legacy files are removed" -->
New data access goes in a query module at `packages/api/src/data/<domain>/queries.ts`
as plain exported functions. Do not add repository classes. The remaining
`*Repository` classes in `services/` are legacy and are being removed. Do not copy them.
<!-- rule: CFG-002 type=referential symbol=isFlagEnabled -->
Feature flags are read through `isFlagEnabled` in `packages/api/src/config/flags.ts`.
<!-- rule: TST-001 type=executable -->
```bash
pnpm --filter api test
```
<!-- rule: BIL-J01 type=judgment owner=payments-team expires=2027-01-31 -->
Prefer boring code in `billing/`: no clever generics, no metaprogramming.
This code gets audited and the auditors are not TypeScript people.The format is `rule: <AREA>-<NNN> type=<type> key=value ...`. The ID lets us cite a rule in lint messages, review comments and commit messages, and lets a script track the rule's adherence over time. A comment applies to the block right after it, which is either a run of contiguous non-blank lines or a single fenced code block.
Step 2: executable claims, run them
Pull every fenced `bash`/`sh` block out of the file, run it in a clean container with a timeout, and fail the build on a non-zero exit. It's boring and it works. It's also the easiest step, which is why most teams stop here.
#!/usr/bin/env python3
"""Executable claims: run every fenced shell block in CLAUDE.md."""
import pathlib, re, subprocess, sys
DOC = pathlib.Path(sys.argv[1] if len(sys.argv) > 1 else "CLAUDE.md")
TEXT = DOC.read_text()
BLOCK = re.compile(
r"(?:<!--\s*rule:(?P<meta>[^>]*)-->\s*\n)?```(?:bash|sh|shell)\n(?P<body>.*?)```",
re.S,
)
TIMEOUT = 300
failures = 0
for m in BLOCK.finditer(TEXT):
line = TEXT[: m.start()].count("\n") + 1
meta = m.group("meta") or ""
if "skip-ci" in meta:
reason = re.search(r'skip-ci="([^"]{10,})"', meta)
if reason:
print(f"skip L{line}: {reason.group(1)}")
else:
print(f"::error file={DOC},line={line}::skip-ci needs a quoted reason (10+ chars)")
failures += 1
continue
try:
r = subprocess.run(
["bash", "-euo", "pipefail", "-c", m.group("body")],
capture_output=True, text=True, timeout=TIMEOUT,
)
ok, out = r.returncode == 0, (r.stdout + r.stderr)[-2000:]
except subprocess.TimeoutExpired:
ok, out = False, f"timed out after {TIMEOUT}s"
if not ok:
failures += 1
print(f"::error file={DOC},line={line}::command block failed\n{out}")
else:
print(f"ok L{line}")
sys.exit(1 if failures else 0)Two kinds of command can't just run in CI, so we handle them explicitly:
- **Destructive commands**, like resetting the local DB or regenerating migrations. If the tool has a dry-run flag, the file documents the dry-run form and the agent gets the real form from the script's `--help`. Example: `pnpm db:migrate --dry-run`.
- **Commands that need secrets.** The exec job gets no secrets, on purpose. These blocks carry `skip-ci` plus a reason that says where they're proven instead: `<!-- rule: E2E-002 type=executable skip-ci="needs STRIPE_TEST_KEY; proven by nightly e2e workflow" -->`. A skip without a reason fails. That one requirement got rid of most of our lazy skips.
Here's the job. Note the triggers: it runs on **every PR**, not only PRs that touch CLAUDE.md. Instruction files drift because the code changes underneath them. Someone renames a script in `package.json` in a PR that never touches CLAUDE.md. If the check only runs when the file changes, it misses the very drift it was built to catch.
name: claude-md-claims
on:
pull_request:
schedule:
- cron: "17 3 * * *" # nightly, catches drift from merged code
jobs:
static:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: sudo apt-get update && sudo apt-get install -y ripgrep
- run: pip install pyyaml
- run: python claims/check_rules.py CLAUDE.md # gate + normative + judgment
- run: python claims/check_refs.py CLAUDE.md # paths + symbols
exec:
runs-on: ubuntu-latest
container: node:20-bookworm # clean box, no cached state, no secrets
timeout-minutes: 20
steps:
- uses: actions/checkout@v4
- run: corepack enable && pnpm install --frozen-lockfile
- run: python3 claims/check_exec.py CLAUDE.mdStep 3: referential claims, stat and resolve
Every path in backticks has to exist. Every symbol named in a rule comment has to resolve to a declaration. We use ripgrep rather than a language server because it finishes in two seconds on a bare runner. If your codebase has lots of exports with the same name, use the language server instead.
#!/usr/bin/env python3
"""Referential claims: every path exists, every symbol= resolves."""
import pathlib, re, subprocess, sys
DOC = pathlib.Path(sys.argv[1] if len(sys.argv) > 1 else "CLAUDE.md")
PATH = re.compile(r"`((?:[\w.@-]+/)+[\w.@-]+\.\w+)`") # `a/b/c.ts`; skips <placeholders>
RULE = re.compile(r"<!--\s*rule:(.*?)-->")
SYMBOL = re.compile(r"symbol=([A-Za-z_$][\w$]*)")
DECL = r"export\s+(default\s+)?(async\s+)?(class|function|const|interface|type|enum)\s+{}\b"
failures = 0
def fail(n, msg):
global failures
failures += 1
print(f"::error file={DOC},line={n}::{msg}")
for n, line in enumerate(DOC.read_text().splitlines(), 1):
for p in PATH.findall(line):
if not pathlib.Path(p).exists():
fail(n, f"path does not exist: {p}")
rule = RULE.search(line)
for sym in SYMBOL.findall(rule.group(1) if rule else ""):
r = subprocess.run(
["rg", "-q", "--type", "ts", "-e", DECL.format(re.escape(sym)), "."],
)
if r.returncode != 0:
fail(n, f"symbol not declared anywhere: {sym}")
sys.exit(1 if failures else 0)On its first run it caught the moved auth middleware. The file still said `packages/api/src/middleware/requireAuth.ts`, but a refactor months earlier had moved it under `packages/api/src/http/auth/`. Every agent session that touched auth had been hitting that wall and working around it. It also caught a couple of dead script paths. All of it was loud rot, all of it was cheap, and we fixed it in one afternoon.
It did not catch line 14. The example path in line 14, `services/orders/OrderRepository.ts`, existed and resolved fine. A legacy file is still a real file.
Step 4: normative claims, the step that would have caught line 14
Every convention has to point at something that enforces it: an `enforce=` lint rule, an `example=` canonical file, or ideally both. Without one, the rule is just an opinion written as a bullet point. Here's SVC-003 as a local ESLint rule:
// eslint-rules/no-repo-class.js
module.exports = {
meta: {
type: "problem",
messages: {
repo: "SVC-003: no repository classes. Add plain functions to packages/api/src/data/<domain>/queries.ts.",
},
},
create(context) {
return {
ClassDeclaration(node) {
if (node.id && /Repository$/.test(node.id.name)) {
context.report({ node, messageId: "repo" });
}
},
};
},
};
// eslint.config.js (excerpt)
// {
// plugins: { local: localRules },
// rules: { "local/no-repo-class": "error" },
// ignores: [ ...require("./claims/legacy-baseline/SVC-003.json") ], // may only shrink
// }Legacy files go into a baseline ignore list, and a small check makes sure that list never grows. Next comes the step almost everyone skips: **measuring adherence**. How much of the code this rule covers actually follows it today?
Line 14 would have failed this check right away. The old repository pattern was still the majority in `services/`. So the rule wasn't describing a convention. It was describing an unfinished migration and presenting it as settled fact. When you ask an agent to follow a rule, it tends to copy whichever pattern it finds more of. A rule at 60% adherence isn't a rule yet. It's a migration in progress, and the file has to say so and say which pattern wins.
Adherence is defined per rule in a small YAML file, with an old pattern, a new pattern and a scope. A file counts as adherent if it matches the new pattern and not the old one.
# claims/adherence.yml
SVC-003:
scope: "packages/api/src/**/*.ts"
old: 'class \w+Repository\b'
new: 'from "@/data/[a-z-]+/queries"'
LOG-002:
scope: "packages/*/src/**/*.ts"
old: 'from "@/lib/legacyLogger"'
new: 'from "@/observability/log"'Next is `check_rules.py`. It does three jobs: the classification gate (every line has a type), the normative checks (an enforcer exists and adherence is at or above the floor) and the judgment checks (owner and expiry). It's one script because all three parse the same comments.
#!/usr/bin/env python3
"""Gate + normative + judgment checks for CLAUDE.md rule comments."""
import datetime, pathlib, re, subprocess, sys, yaml
DOC = pathlib.Path(sys.argv[1] if len(sys.argv) > 1 else "CLAUDE.md")
RULE = re.compile(r"^<!--\s*rule:\s*([A-Z]+-J?\d{2,3})\s+(.*?)\s*-->$")
TYPES = {"executable", "referential", "normative", "judgment"}
ADH = yaml.safe_load(open("claims/adherence.yml"))
ESLINT = pathlib.Path("eslint.config.js").read_text()
TODAY = datetime.date.today()
failures, seen = 0, set()
def fail(n, msg):
global failures
failures += 1
print(f"::error file={DOC},line={n}::{msg}")
def files(pattern, glob):
r = subprocess.run(["rg", "-l", "--glob", glob, "-e", pattern, "."],
capture_output=True, text=True)
return set(r.stdout.split())
def check_rule(n, rid, attrs):
t = attrs.get("type")
if t not in TYPES:
return fail(n, f"{rid}: type must be one of {sorted(TYPES)}")
if t == "normative":
enf, ex = attrs.get("enforce", ""), attrs.get("example")
if not enf and not ex:
fail(n, f"{rid}: normative rule needs enforce= or example=")
if enf.startswith("eslint:") and enf.split(":", 1)[1] not in ESLINT:
fail(n, f"{rid}: lint rule {enf} not registered in eslint.config.js")
if ex and not pathlib.Path(ex).exists():
fail(n, f"{rid}: example file missing: {ex}")
if rid not in ADH:
return fail(n, f"{rid}: no adherence definition in claims/adherence.yml")
a = ADH[rid]
old, new = files(a["old"], a["scope"]), files(a["new"], a["scope"])
total = len(old | new)
score = len(new - old) / total if total else 1.0
floor = float(attrs.get("adherence-min", 0.9))
print(f"{rid}: adherence {score:.0%} ({len(new - old)}/{total} files)")
if score < floor:
fail(n, f"{rid}: adherence {score:.0%} < {floor:.0%}. This is a migration, not a rule. "
"Say so in the line, list the legacy pattern as do-not-copy, or finish it.")
if t == "judgment":
if not attrs.get("owner"):
fail(n, f"{rid}: judgment line needs owner=")
try:
exp = datetime.date.fromisoformat(attrs.get("expires", ""))
if exp < TODAY:
fail(n, f"{rid}: expired {exp}. Owner {attrs.get('owner')} must re-sign or delete.")
except ValueError:
fail(n, f"{rid}: judgment line needs expires=YYYY-MM-DD")
tagged = in_fence = False
for n, raw in enumerate(DOC.read_text().splitlines(), 1):
s = raw.strip()
if s.startswith("```"):
if not in_fence and not tagged:
fail(n, "untyped code block")
in_fence = not in_fence
continue
if in_fence:
continue
if not s:
tagged = False
continue
if s.startswith("#"):
continue
m = RULE.match(s)
if m:
rid = m.group(1)
if rid in seen:
fail(n, f"duplicate rule id {rid}")
seen.add(rid)
attrs = dict(kv.split("=", 1) for kv in re.findall(r'[\w-]+=(?:"[^"]*"|\S+)', m.group(2)))
check_rule(n, rid, attrs)
tagged = True
continue
if not tagged:
fail(n, "untyped line: add a rule comment or delete it")
sys.exit(1 if failures else 0)About that 0.9 default: we chose it, we didn't derive it. Below roughly 90%, an agent searching for examples has a real chance of landing on the legacy pattern first. Above it, the lint rule and the example file outweigh the stragglers. Pick your own floor, but pick one. When a rule drops below it, you have three honest options:
- **Finish the migration.** This is the best option, and it usually won't happen this week.
- **Rewrite the line as a migration.** Name the legacy pattern explicitly as do-not-copy and point at the lint rule that blocks it in new code. Then lower `adherence-min` and add a note saying why. The SVC-003 text above is this version.
- **Delete the rule.** If nobody is going to enforce it, the file shouldn't claim it.
The original line 14 said something false about the codebase and gave no hint that the claim was disputed. What actually changed the agent's behaviour was rewriting the line as a migration and labelling the old pattern as legacy. The lint rule is a backstop for cases where the wording doesn't work.
Step 5: judgment claims, keep them but sign them
Some guidance can't be linted, and it shouldn't have to be. "Prefer boring code in `billing/`" is one of the most useful lines in our file. It sets a tone, and an agent can reason with its rationale (auditors read that code). We keep lines like this. They just need a name and a date:
<!-- rule: BIL-J01 type=judgment owner=payments-team expires=2027-01-31 -->
Prefer boring code in `billing/`: no clever generics, no metaprogramming.
This code gets audited and the auditors are not TypeScript people.When a line expires, CI fails on every PR until someone deals with it. The owner either re-signs with a new date or deletes the line. Most of the time they re-sign, and that's fine, because it means someone read the line this quarter and still agrees with it. Deletion is more common than you'd expect, though. People take a line seriously when it's their name blocking the build. We use a 3–6 month window. If it's shorter, re-signing becomes a reflex. If it's longer, nobody remembers why the line was written.
Judgment IDs get a `J` in the number (`BIL-J01`), so grepping for `-J` lists every unprovable claim in the file. That list should stay short. If it keeps growing, the team is writing down opinions when it should be turning them into checks.
The merge gate: untyped lines don't merge
The last rule is the strict one: **any non-heading line without a type fails CI.** Old lines get no exemption. On the first run, most of the failures were untyped lines, and a big chunk of the file turned out to be opinion with no owner and no type. Some lines were years old, with a dozen authors in git blame. Some contradicted each other. Some described tools we'd dropped. A few were obviously one person's reaction to one bad PR.
We tried classifying all of it and gave up after an hour, because we kept defending lines nobody had read in a year. So we flipped it around: any untyped line that nobody volunteered to own by the end of the meeting got deleted. If a deletion turned out to matter, an agent PR would show us, and we'd add the line back with a type. A few lines did come back that way. Most never did.
The gate matters more for future edits than for the cleanup. Without it, the file turns back into an untyped scratchpad within a quarter, because adding one more bullet is always easier than writing a lint rule.
What changed, and what we can't claim
The file got a lot shorter. That's easy to measure, and I'm confident it helped. The agent has less to weigh, and fewer lines contradict each other.
We tracked two things more carefully:
- **Change-request review comments on agent-authored PRs.** We pulled them from the GitHub API, kept only PRs opened by our agent bot accounts, and compared similar windows before and after. The count went down. I'm not putting a percentage on it. The windows were short, we upgraded models in the same period, and the team was getting better at writing task prompts at the same time. I can't separate those effects cleanly, so I won't pretend to.
- **Reintroductions of the repository pattern.** These dropped to zero once SVC-003 had its lint rule, but that's guaranteed, because the lint rule blocks the merge. It tells you nothing about the instruction file. A more honest signal is how often agent branches *failed* the `local/no-repo-class` lint check before review, i.e. how often the agent wrote the legacy pattern in the first place. That dropped noticeably after we rewrote line 14 as an explicit migration. I think that's the best evidence we have that the wording mattered on its own, apart from the enforcement.
The thing I'm surest of isn't really about agents. We now know which lines in the file are true. Before, nobody would have bet on any given line. Now every line either passed a check last night or has a person's name on it.
Copy this: a minimal starter
Here's the smallest version you can drop into a repo this week:
CLAUDE.md # every block preceded by <!-- rule: ... -->
claims/
check_rules.py # gate + normative (enforcer, adherence) + judgment (owner, expiry)
check_refs.py # paths exist, symbol= resolves
check_exec.py # run fenced shell blocks, skip-ci requires a reason
adherence.yml # old/new/scope per normative rule
legacy-baseline/ # lint ignore lists that may only shrink
.github/workflows/claude-md-claims.yml # runs on every PR + nightlyThe full rule-ID grammar:
<!-- rule: <AREA>-<NNN> type=executable [skip-ci="<reason, 10+ chars>"] -->
<!-- rule: <AREA>-<NNN> type=referential [symbol=<Name>] -->
<!-- rule: <AREA>-<NNN> type=normative enforce=eslint:<rule> | example=<path> [adherence-min=0.9] [adherence-note="<why lowered>"] -->
<!-- rule: <AREA>-J<NN> type=judgment owner=<team-or-person> expires=YYYY-MM-DD -->The rollout order we'd suggest:
- **Day 1:** add `check_refs.py` and `check_exec.py`. They'll catch your loud rot. Fix it and take the easy win.
- **Day 2:** go through the file and find every line that states a convention. For each one, write its `old`/`new` patterns in `adherence.yml` and run the adherence check. Any rule under your floor is a migration you'd forgotten about. This is where you'll find your own line 14.
- **Day 3:** turn on the classification gate and hold one meeting. Lines nobody owns by the end of it get deleted.
- **After that:** add one question to your PR template for any change to CLAUDE.md or AGENTS.md.
<!-- .github/pull_request_template.md (excerpt) -->
### Touches CLAUDE.md / AGENTS.md?
For every added or changed line: **which type is this line, and what proves it?**
- executable → it runs in CI
- referential → the path/symbol resolves
- normative → link the lint rule or example file, and paste the adherence %
- judgment → owner and expiry dateAsking that question in review is the real change. Agents take your instruction file literally. So every line in it needs either a CI check that proves it or a person who answers for it.