Cold-start agent smoke test

Why this matters

Your next hire will spend their first week asking questions that your AGENTS.md should already answer. Your coding agents run into the same gaps today, and each gap costs you one bad PR per session. You can find most of them in a single afternoon before the human starts. Give a ticket to a fresh agent and watch where it goes wrong.

The cold-start run that replayed week one

Before our last backend hire started at PhoenixDX, I ran an experiment. I took a small ticket from the backlog and gave it to a fresh Claude Code session. It had no memory and got no hints from me, just the repo and our AGENTS.md. I wrote down every wrong turn it took. Then I dug up the Slack questions our previous hire had asked in their first week and put the two lists side by side. Nearly every line had a match.

• ▹Failure 1: The agent ran the test suite, hit database connection errors and decided to mock the database. The hire asked: "Do I need a local Postgres, or is there a docker compose thing?"
• ▹Failure 2: The agent read .env.example and made up placeholder values for the payments sandbox key. The hire asked: "Where do I get the sandbox keys?"
• ▹Failure 3: The agent shipped a new endpoint with no feature flag around it. The hire asked: "Should new stuff go behind a flag?"
• ▹Failure 4: The agent branched from main and opened the PR against main. The hire asked: "Which branch do PRs target? Is there a naming rule?"
• ▹Failure 5: The agent picked the old auth helper because it had the most call sites in the repo. The hire asked: "Is legacy-auth still a thing? It's everywhere."
• ▹Failure 6: The agent edited a shared API contract that belongs to another team. The hire asked: "Can I change this schema, or do I need to ask someone first?"

The agent made all six mistakes in about forty minutes. The human made them across five days. Both runs came back to the same root cause: none of it was written down.

The reframe: the agent is a loud new hire

Humans paper over the gaps in a doc. A new hire who can't find the sandbox keys won't file a bug against your docs. They DM someone, copy a teammate's .env, or guess and move on. Each of those is a sensible thing to do, and each one keeps the gap hidden. The next hire hits the same gap. The senior engineer who answered the DM pays a small tax every time and never notices.

An agent can't ask quietly. It fills the gap with a confident guess, and the guess shows up in the PR diff for everyone to see. When that happens, resist the urge to blame the agent and tweak the prompt. The guess tells you a sentence is missing from the shared context. Earlier in this series, in the context-engineering lesson, we saw that an agent can only act on what's in its context window. This lesson turns that rule into a test: if something is missing from the file, both of your readers are missing it.

The ritual, step by step

• ▹1. Pick a real ticket of about two hours. A toy task tests nothing. A two-week epic fails for too many reasons to diagnose. You want something that touches the build, the tests, a convention or two, and at least one boundary with another team's code.
• ▹2. Start a fresh session with zero memory and no human hints. Clear user-level memory, don't paste a context brief and don't nudge it when it stalls. If you skip any of this, you're testing your own head and not the file.
• ▹3. Let it run all the way to a PR. Don't stop it at the first mistake. The later failures, like the wrong base branch or a missing changelog entry, are often the most useful.
• ▹4. Log every wrong assumption as a 'missing sentence': one fact that would have prevented the mistake, written in the exact words you'd put in the file.
• ▹5. Add the sentence to AGENTS.md, not to the prompt. A prompt is gone when the session ends. The file is read by every agent and every human who comes after you. If a fix only works in a prompt, it doesn't count.
• ▹6. Rerun in a fresh session until it ships clean. Then run once more on a different ticket, because a single clean pass can be luck.

Cold-start prompt (paste as-is, nothing else):

You are new to this repository. Implement ticket PDX-1234 using only
what is in this repo and its docs. Open a PR when done. If you have
to assume something that is not written down, say so explicitly in
the PR description under 'Assumptions'.

---
Missing-sentence log (one line per failure):

| # | What the agent did            | Missing sentence                                  | Category    |
|---|-------------------------------|---------------------------------------------------|-------------|
| 1 | Mocked the DB in tests        | "Run `make db-up` before tests; never mock Postgres." | Environment |
| 2 | PR targeted main              | "All PRs target `develop`; main is release-only."   | Convention  |

The 'Assumptions' line is the most useful part of that prompt. The agent lists its own guesses for you, which takes most of the work out of step 4.

What the failures usually look like

After a few runs, the missing sentences fall into four categories. Once you know them, you can write most of the file before you run the test. They also tell you which sections your new hire will lean on hardest.

• ▹Environment and secrets: anything that lives outside the repo, such as local services, sandbox credentials, and which values in .env.example are real. At PhoenixDX, the payments sandbox key comes from the team vault, and the agent made one up. The missing sentence was: "Sandbox keys live in the team vault under payments/dev; never commit or fabricate them, and stop and ask if you can't access the vault."
• ▹Unwritten conventions. Flags, naming, branch strategy and PR shape all fall here, along with the other rules reviewers enforce from memory. Our rule is that every new user-facing endpoint ships behind a flag that's off by default. Nobody had written it down because 'everyone knows'. Missing sentence: "New user-facing behaviour must be behind a flag in flags.ts, default off."
• ▹'We don't do it that way anymore' history. The old way is still in the codebase, and often there's more of it than the new way. Agents follow the patterns they see most, so they copy whichever one shows up most often. Our legacy auth helper had far more call sites than its replacement, so the agent picked it. Missing sentence: "legacy-auth is deprecated; use auth/session for all new code. Old call sites are being migrated; don't copy them."
• ▹Ownership boundaries: what you can touch, what needs another team's review, and what you must never edit. The shared contracts package belongs to the platform team, and any change there ripples into other services. Missing sentence: "Do not edit packages/contracts in feature PRs. Open a contract-change issue instead."

In my experience, teams underestimate history the most. For an agent, the codebase is the strongest signal it has. If the old pattern dominates the code and the file is silent, the agent will copy the old pattern.

One file, two readers

Write one file and point every tool at it. Codex and Cursor read AGENTS.md directly. For Claude Code, CLAUDE.md can be a single line that imports @AGENTS.md. If you keep a second copy for humans, it will drift from the agent version, and usually sooner than you'd expect. Order the file by how often each part gets used:

• ▹Commands first: setup, test, lint and run, ready to paste as written. Both readers need these within ten minutes.
• ▹Then invariants, the rules that must always hold: flags, branch targets, forbidden directories and security constraints. Write short imperative sentences, each one something a reviewer can check.
• ▹History third. Name what's deprecated and what replaced it so nobody copies the old way. Keep it to one line per item, not an essay.
• ▹Keep the whole thing short and opinionated. Every line costs the agent context tokens and costs the human attention. If a sentence wouldn't change anyone's behaviour, cut it. Link to the deeper docs instead of pasting them in.

# AGENTS.md

## Commands
- Setup: `make bootstrap` (needs Docker running)
- DB for tests: `make db-up` — never mock Postgres in tests
- Test: `make test` | Lint: `make lint` | Run: `make dev`

## Invariants
- PRs target `develop`. Branch names: `<type>/<ticket>-<slug>`.
- New user-facing behaviour goes behind a flag in `src/flags.ts`, default off.
- Secrets come from the team vault. Never fabricate or commit them.
  If you can't get one, stop and say so.
- Do not edit `packages/contracts/` in feature PRs; open a contract-change issue.

## History (don't copy the old way)
- `legacy-auth` → use `auth/session`. Old call sites are mid-migration.
- REST handlers in `api/v1/` are frozen; new endpoints go in `api/v2/`.

## Done means
- Tests and lint pass, flag added, PR description lists assumptions.

## More
- Architecture: docs/architecture.md
- Humans: see docs/onboarding/people.md (not for agents)

Where the smoke test lies

A pass tells you the written-down knowledge is complete enough to ship one small change. It doesn't tell you a human is onboarded, for three reasons. First, an agent reads dozens of files a minute and can work around a vague sentence that would cost a human a day. Second, an agent doesn't need to know why the system looks the way it does, but the human who owns it in six months does. Third, an agent can't tell anyone who to ask, because a team's social graph doesn't live in markdown.

Put a People section in a separate file that AGENTS.md links to but agents don't load, such as docs/onboarding/people.md. Keeping it separate saves context tokens, keeps social information out of agent prompts, and lets the file change on its own schedule. List roles, owners and team channels. Leave out personal contact details. Those belong in your company directory, not in the repo.

# People (for humans; agents don't need this)

## Who owns what
- Payments service: Payments team (#team-payments). Reviewer rota in channel topic.
- packages/contracts: Platform team (#platform). Contract changes need their approval.
- CI and deploys: Platform on-call, see the on-call rota.

## Who to ask about X
- "Why is it built like this?": the tech lead for that service
- Product intent behind a ticket: the PM listed on the ticket
- Anything that feels unsafe: say it in #eng, nobody minds

## Three questions to ask a person in week one
1. What broke in production most recently, and what did we change because of it?
2. Which part of the codebase do people avoid touching, and why?
3. Whose review do I need for what, and what do they care about most?

No agent can answer those three questions. Each answer also starts a working relationship, which no doc can do. Make sure your new hire asks them in person.

Making it a sprint habit

• ▹Run it every sprint on one real ticket. One person can do it in an afternoon, and the agent does most of the work.
• ▹Run it after big refactors. A refactor turns today's conventions into history overnight, so the file goes stale fastest right after one.
• ▹Run it before every new hire starts, ideally a week or two ahead, so the fixes are in place on their first day.
• ▹Track 'missing sentences found' per run. It's a cheap measure of onboarding debt. Expect it to trend toward zero, with spikes after refactors and infrastructure changes. A spike means the file fell behind the code.
• ▹Don't turn the metric into a target. Once people start picking easy tickets to make the run pass, the number improves while the file stays as incomplete as it was. Rotate who picks the ticket.
• ▹Give the file a length budget. If every fix only adds lines, the file grows until both readers skim it. Each time you add a sentence, check whether an old one can go.

Bridge to tomorrow

Once the file is good enough for an agent to ship a clean PR, the new hire's first job is not to read it. Their first job is to run an agent on a real ticket with that file and catch the agent when it's wrong. It's a fast way to learn a codebase, and every catch they make is another sentence the file is missing. Reviewing an agent's work as a newcomer is a separate skill, and that's tomorrow's lesson.

Extend your knowledge

• ▹Read the AGENTS.md open format (agents.md) and check which of your tools read it natively. For Claude Code, read the memory docs on CLAUDE.md and @-imports so you can keep a single source file.
• ▹Reread Anthropic's engineering post 'Effective context engineering for AI agents' with your own AGENTS.md open next to it. Mark every line that doesn't change behaviour.
• ▹Look at the Diátaxis documentation framework (tutorials, how-to guides, reference, explanation). It explains why commands and invariants belong in AGENTS.md and why architecture explanations should be linked out.
• ▹This sprint, pick one two-hour ticket, run the cold-start prompt from this lesson and log the missing sentences. That count is your onboarding-debt baseline.