Cold dogfooding

Day 23: Cold Dogfooding — Why Your Team's Usage Doesn't Tell You If Your Agent Works

Your team uses its AI agent every day and it almost always works. That's the number you trust most, and it's probably the least reliable one you have. When the interface is natural language, everything you already know about the codebase goes into the prompt. A new user doesn't have any of that knowledge, and the product doesn't supply it.

1. The afternoon the numbers split

We had an internal coding agent that the team used all the time for migrations, endpoint scaffolding and service refactors. As far as we were concerned, it worked. Then a new engineer joined and I sat in on their first session. They tried three tasks we'd have called routine, and all three failed. One change went into the wrong service. One ignored our naming conventions. One "fixed" a bug by rewriting part of a legacy module that nobody is supposed to touch.

Once we tracked it properly, the gap was obvious. Our team completed about 90% of tasks successfully. People new to the agent completed about 40%. The model, the repo and the tasks were the same in both cases. The only difference was who was typing.

2. The first wrong guesses: the model, then the person

First we blamed the model: a bad day, maybe an unlucky sample. Then we blamed the new hire, which was worse: maybe they just weren't good at prompting yet. Both explanations were comfortable because neither required us to change anything.

What changed our minds was simple. We put the new hire's prompt next to one of ours for the same task:

NEWCOMER:
  Add a field for customer tier to the billing flow.

INSIDER (same task, written without thinking about it):
  Add `customer_tier` to the billing flow in services/billing/.
  Follow the snake_case + Pydantic model pattern in models/invoice.py.
  Add an Alembic migration. Don't touch legacy/payments_v1 —
  it's frozen. Run the billing test suite only.

Nobody on the team had ever decided to include file paths, naming rules or "don't touch the legacy module." We typed them out of habit, and those clauses did most of the work. The agent wasn't succeeding for us. We were succeeding through it.

3. Name the bias: author-blindness

Author-blindness is my name for what happens when the people who build a product can't see it the way a user does, because their own knowledge fills the gaps without them noticing. It's closely related to what psychologists call the curse of knowledge. Every kind of software has it, but agents make it much worse.

• ▹In traditional software, the interface is fixed. When a builder clicks "Export" and a user clicks "Export," the same code runs. Your dogfooding tests the product.
• ▹With an agent, the prompt is the interface. Everything the builder knows about repo layout, conventions, landmines and which phrasing works ends up in every input.
• ▹That means internal success measures product quality and operator fluency together, and you can't separate the two. Without realizing it, you're testing yourself.
• ▹It also drifts. The longer the team uses the agent, the better they get at working around its gaps without noticing. Internal metrics go up while the product stays the same.

4. Where this fits in the series

Earlier in the series we separated what users say from what users do. This is the same trap, aimed at ourselves. Our own usage was the least trustworthy data we had, because every one of those users had also built the thing.

5. The practice: cold dogfooding

Cold dogfooding means testing your agent with people who have no context on how it works, under rules that stop insiders from coaching them to success. This is how we ran it:

• ▹Rotate in cold users. Use engineers who have never seen the agent's system prompt, tool definitions or internal docs. New hires are ideal, and engineers from another team work well too. Coaching someone even once means they aren't cold anymore, so keep rotating.
• ▹Ban prompt tweaks. Nobody rewords anything, nobody says "try it like this," and nobody leans over a shoulder. The first prompt the user writes is the test. Once you coach, you're testing yourself again.
• ▹Use real tasks, not demo tasks. Pull them from the actual backlog. Demo tasks are the ones the team already knows will succeed.
• ▹Log every failure exactly as it happened: the original prompt, what the agent did and what the user expected. Don't summarize it, and don't write it off as "user error."
• ▹Keep a hidden-knowledge list. For every failure, an insider writes down what they would have typed that the newcomer didn't. This list is the most valuable thing the exercise produces.

# cold-dogfood-log.yaml — one entry per failed attempt
- date: 2026-03-04
  user: cold-07            # anonymised; no names in the log
  task: "Add customer tier to billing"
  prompt_verbatim: "Add a field for customer tier to the billing flow."
  agent_did: "Edited services/checkout/, not services/billing/"
  expected: "Change in billing service + migration"
  hidden_knowledge:
    - "Billing lives in services/billing/, checkout is a different service"
    - "legacy/payments_v1 is frozen"
  fix_type: product        # product | onboarding | wont_fix
  fix_ref: "repo-discovery step"

6. What the hidden-knowledge list turned into

The rule was that nothing on the list could stay in our heads. Every item went into the product, went into onboarding, or was explicitly marked as a known limitation. Most items fell into three product groups, and onboarding covered the rest:

• ▹Repo discovery, which closed the most gap. Most insider prompts began with file paths, so we made the agent find the relevant service and the patterns around it before writing any code. Most of the "edited the wrong place" failures went away.
• ▹Defaults and guardrails. Naming patterns, which test suite to run and which modules are frozen became project-level context loaded on every run, similar to a CLAUDE.md or rules file. "Don't touch legacy" no longer depended on someone remembering to type it.
• ▹Clarifying questions. When a request was ambiguous (which tier? which flow?), the agent now asks before it acts. That makes it slower on simple requests and much safer on vague ones, and vague requests were what cold users actually wrote.
• ▹Onboarding for whatever was left. Some knowledge really did belong in a person's head. It went into a short "how to brief the agent" guide that every new user gets, so it stopped depending on word of mouth.

Repo discovery and loaded project context delivered the biggest gains, because they covered the knowledge insiders typed most often without noticing. Clarifying questions came second. Onboarding docs helped least, and that's informative: if the fix is "users should know X," the product still hasn't taken X on.

7. The uncomfortable rule: insider numbers don't count

We added one rule that the team pushed back on: insider success rates are not reported in product reviews. Only cold numbers count.

• ▹"We're real users too." That's true. You're also the only users who never need onboarding, and the smallest group the tool will ever have.
• ▹"Cold numbers are noisy, and the sample is small." Also true, but a small honest sample beats a large biased one. Report the sample size and keep collecting.
• ▹"It makes the agent look worse than it is." It shows the agent as it actually is for everyone outside the team, and that's the product.

The rule was worth the friction because it changed what we worked on. While the 90% number counted, we tuned prompts and added tools for power users. Once only the cold number counted, we worked on discovery, defaults and clarification, which help the next hundred users. The metric we reported ended up setting the roadmap.

When to use cold dogfooding (and when not)

• ▹Use it for any natural-language or agent interface, for any internal tool that will go beyond the people who built it, and before every "it's ready for the wider org" decision.
• ▹Use it when internal adoption looks great but people outside the core team don't stick with the tool.
• ▹Don't make it your only signal for expert-only tools that only the builders will ever use. Even so, run it once, because teams change.
• ▹Don't treat it as usability testing for UI polish. The point is to find out what the product silently assumes the user already knows.

Common pitfalls

• ▹Coaching through body language. Sighing, pointing and "hmm, maybe add the path" are all coaching. If you have to, sit where you can't see the screen.
• ▹Reusing the same cold users. After two sessions they've picked up your hidden knowledge, and they're insiders now.
• ▹Filing failures as "prompting skill issues." When newcomers keep failing in the same way, the product is missing something.
• ▹Fixing everything with documentation. Docs are where hidden knowledge ends up when nobody wants to change the product.
• ▹Logging people instead of prompts. Keep the logs anonymous. You're evaluating the tool, not the new hire.

Day 23 exercise

Give your AI tool, or any internal tool, to someone with zero context for 30 minutes along with a real task. Stay silent. Write down everything you wanted to say: every path, every "oh, you have to...", every "don't click that." That list is your backlog. Sort each item into product, onboarding or won't-fix, then pick the top three to build this sprint.

Extend your knowledge

• ▹Read Steve Krug's 'Rocket Surgery Made Easy' for a lightweight, do-it-yourself way to run small observation sessions, then adapt it to agent tasks.
• ▹Look at how coding agents handle project-level context files, such as CLAUDE.md in Claude Code or rules files in Cursor. They're the natural home for hidden knowledge.
• ▹Go back to the series lesson on 'what users say vs. what users do' and run the same comparison on your team's own agent usage logs.
• ▹Build a small eval set from your cold-failure log, with newcomer prompts kept verbatim and their expected outcomes. Re-run it on every agent change so the cold number works as a regression test.