Kill criteria for AI pilots

Day 30: Write the Kill Criteria Before the Demo

Your AI agent pilot isn't going to fail. It will keep running quietly, eating a senior engineer's afternoons, and every month there will be a decent reason to give it one more month. If nobody agreed up front on when to stop, it simply carries on.

Today's lesson is about one habit. Before an agent's first demo, write down the conditions under which you will shut it down. Base those conditions on human cost, not accuracy. Put a date on them. Then get someone other than the builder to sign.

1. The demo room, and nine months later

I've lived this one. We ran an internal demo of a coding agent. Someone pasted in a live bug from the tracker. The agent read the stack trace, found an off-by-one in a pagination helper, wrote the fix and a test, and opened a PR. About four minutes. The room lit up.

A few minutes later someone asked how it would cope with our legacy billing module. I heard myself say, "With better context it should be fine." The agent hadn't done a single piece of real work yet, and I was already defending it.

Nine months later it was still a pilot. It got roughly 60% of tasks right. A senior engineer reviewed every PR it opened, because nobody could tell in advance which 40% would be wrong. Nobody ever called it a failure. It never paid off either.

2. The autopsy: nothing ever failed

Looking back, I couldn't find a single bad week. Every week came with a reasonable explanation:

• "A new model is coming next month. It'll handle the harder cases."
• "We haven't tuned the system prompt for this repo yet."
• "It needs better context. Let's add the architecture docs to retrieval."
• "The eval set isn't representative. Let's rebuild it."
• "The reviewers are being too strict. Let's calibrate."

Taken one at a time, each of those is fine. Stacked together, they kept a dead project walking. I call this a zombie pilot. It keeps moving and keeps using resources, and nobody can say it's working. Two well-known biases keep it alive, and agents add a third problem of their own.

• Anchoring. The strong demo becomes the reference point, so every bad week reads as a deviation from "what it can really do" when it's actually evidence of what it does.
• Sunk cost. Each month of prompt work, retrieval tuning and eval building makes stopping feel more wasteful. The money already spent turns into the argument for spending more.
• The agent-specific one: there's always another lever. A new model, a prompt change, more context, a different tool. You can always believe the fix is one step away.

3. Why the demo is the worst time to judge

Demo day is when you're most confident and have the least evidence. Three reasons:

• You picked the task. Demo tasks are well-scoped, self-contained, and sit in a corner of the codebase the agent handles well. Real work doesn't get chosen that way.
• You saw one good run, and that hides the variance. LLM agents are non-deterministic, so the same prompt can succeed on Tuesday and go sideways on Wednesday. One run shows what the agent can do. It says nothing about what it usually does.
• Everyone in the room wants it to work. Leadership has an AI story to tell, the engineers are curious, and you built the thing. Nobody has a reason to play the sceptic.

So write the kill criteria before the demo, while you're still unsure. Once the room is excited, you'll set thresholds the agent can step over without trying.

4. The metric you actually need: human cost

Most agent pilots track accuracy: pass rate, task success, eval score. That's the wrong number to decide on, because it ignores what the errors cost the people around the agent.

Here's the part people miss. When you can't see the errors coming, the error rate hits every output. If 15% of outputs are wrong and nothing tells you which ones, someone has to review all of them carefully. You pay for review on 100% of the work, not on the 15%.

The basic comparison per task:

Manual cost per task      = M

Agent cost per task       = R  +  p × F  +  C
  R = review time (paid on EVERY output, by whoever can judge it)
  p = fraction of outputs that need correction
  F = fix time when it's wrong (often > M: you must first
      understand the agent's wrong approach, then undo it)
  C = compute/API cost, converted to minutes of engineer time

The agent pays off only if:   R + p × F + C  <  M

Illustrative example (made-up numbers, plug in your own):
  M = 45 min   mid-level engineer does the task by hand
  R = 25 min   senior must review every PR closely
                -> weight senior time ~1.5x  => ~37 mid-level-equivalent min
  p = 0.15     85% "accurate"
  F = 60 min   untangling a plausible-but-wrong fix
  C = ~3 min   tokens, CI reruns

  Agent: 37 + 0.15 × 60 + 3  = 49 min
  Manual:                      45 min

  85% accurate, and still a net loss.

The formula leaves out two costs. Track them anyway:

• Bottleneck cost. If only one senior can review the agent's output, the pilot moves at the speed of that person's calendar, and their own work slows down too.
• Review fatigue. Reviewing plausible-looking code carefully is tiring. A few weeks in, reviewers may start approving PRs after a quick skim. Your measured correction rate goes down while real defects go up. A metric that looks better can mean things have got worse.

To bring human cost down, you usually have to cut R, not just p. You can make outputs machine-verifiable with tests, type checks and property checks. You can narrow the task class until mistakes are obvious. Or you can have the agent show its evidence, so review becomes a quick check rather than working the whole problem out again. An agent that's right 70% of the time and obviously wrong when it fails can beat one that's right 90% of the time and subtly wrong when it fails.

5. The template: paste this into your design doc

Kill criteria are three or four thresholds you commit to in advance, each with a date. Put them in the spec or design doc, next to the architecture, where the people building the agent will keep seeing them. Keep them off slides. Slides get rewritten for every audience.

## Kill Criteria — <Agent name> pilot

Written: <date, BEFORE first demo>
Review date: <date, typically 4–6 weeks after pilot start>
Task class in scope: <e.g. "dependency bumps + failing-test fixes in services/api">
Baseline: manual time per task = <M> min (measured on last <N> tasks by hand)

We stop the pilot if, on the review date, ANY of these is true:

1. Human-correction rate > <X>% of agent outputs over the last 2 weeks
   (correction = any human edit beyond formatting before merge, or a revert within 14 days)

2. Median review time > <Y>% of manual time M
   (measured from PR open to approval, active reviewer time, not wall clock)

3. Fully loaded cost per merged PR > cost of a human doing the same task
   (review + fix + compute, using the formula in section <ref>)

4. Only one person can competently review the output
   (bus factor of the review = 1 → the pilot cannot scale, stop or re-scope)

Extension policy:
- ONE extension of <2–4> weeks is allowed.
- It must name the specific change expected to move a specific criterion
  (e.g. "Model <X> release, expected to cut correction rate from 30% to <20%").
- If the criteria still fail after the extension, the pilot ends. No second extension.

What "stop" means: agent disabled for this task class; learnings written up;
re-proposal allowed only as a NEW pilot with NEW criteria.

Signed:
- Builder / pilot owner: ________
- Accountable sponsor (EM or product owner): ________
- Independent signer (someone who does NOT benefit from the pilot continuing,
  e.g. a senior reviewer who carries the review load, or a peer EM): ________

Picking the numbers:

• Measure manual time before the pilot starts. Without a baseline, you'll end up comparing the agent with your memory of the demo.
• Set thresholds that make the pilot clearly worth it, not ones it can merely survive. If review time at 90% of manual time counts as "fine", the pilot isn't worth running.
• Define "correction" precisely, and include reverts after merge. Otherwise review fatigue will flatter your numbers.
• The independent signer matters most. Whoever carries the review load is the best check on the builder's optimism.

6. The hard part: actually keeping to it

Writing the criteria takes an hour. The hard part is the review date, when the numbers miss and someone sensible says, "The next model drops in three weeks. It'd be a waste to stop now."

Sometimes they'll be right, which is why the template allows one extension. It's also why that extension has to be specific and agreed in advance:

• Name the change. A particular model release, a particular context change, a narrower task scope. "General improvements" doesn't count.
• Say which metric it should move, and by how much.
• Fix the date. When it arrives, you apply the criteria again, and nobody gets to renegotiate them.
• If the criteria fail a second time, the pilot stops. You made that call back when you could think about it calmly.

Stopping on schedule means the process did its job. It doesn't mean the team failed. Say this out loud, because engineers easily hear "we stopped your project" as "you failed." What actually happened is that you spent a bounded amount of time and learned something specific: which task classes don't work, what review really costs, where the model is weak. You also freed up senior time. A stopped pilot with a clear write-up is often the thing that gets the next pilot approved, with a better scope.

Watch for these moves when the review date comes:

• Changing the metric. "Correction rate missed, but look how much faster the first drafts are!" If draft speed mattered, it belonged in the criteria.
• Changing the scope after the fact, by quietly dropping the hard tasks from the measurement until the numbers pass.
• Turning the pilot into a permanent "experimental" tool that still uses reviewer time but never has a review date.

7. Closing the course

For 29 days this series has been about building the right thing: understanding users, measuring outcomes, scoping well. Day 30 is the other half. A product mindset also means deciding up front when you'll stop. That matters more in AI work than anywhere else, because there's always one more model, one more prompt, one more tweak.

Your assignment: before your next agent demo, write a sentence that starts with "We stop if…" and get someone who isn't you to sign it.

Extend your knowledge

• Annie Duke, "Quit: The Power of Knowing When to Walk Away". Popularised the term "kill criteria" and explains why quitting decisions should be made in advance, with dates and conditions attached.
• Daniel Kahneman, "Thinking, Fast and Slow". The chapters on anchoring and the planning fallacy explain why the demo bends your judgment.
• Hamel Husain, "Your AI Product Needs Evals". How to build task-specific evals that reflect real work, so your criteria rest on representative tasks and not on demo picks.
• Google's SRE book, chapter 3, "Embracing Risk" (error budgets). A mature engineering example of a pre-agreed threshold that triggers an agreed response, and a good model for holding to kill criteria.