Owning the definition of done

Day 30: Own the Definition of Done

Your agent can write code faster than your whole team can read it. So being able to produce code no longer sets you apart. Deciding what counts as finished does, because every piece of work, human or agent, gets measured against that decision.

1. The dashboard moment

At PhoenixDX there was a week when the merge dashboard showed our coding agents had opened and merged more PRs than all the human engineers put together. Nobody mentioned it. The number just sat there on the screen. That Friday I went back through my own week to work out what I had actually contributed. I didn't like the answer: mostly I'd approved things I hadn't really checked.

It wasn't laziness. There was more work than line-by-line review could keep up with, so I'd started trusting green checkmarks that I hadn't defined. If you use Claude Code, Cursor, Copilot agent mode or something like them, you've probably had a smaller version of this. The diff looks fine and CI passes, so you approve it, without being sure what you've just agreed to.

2. The quiet failure under the throughput

One agent PR shows the problem well. The ticket asked for retry logic on a third-party webhook integration. The agent added retries and wrote unit tests, and every test passed. We merged it. A few days later we found it also retried on a class of client error that should never be retried. A malformed payload got sent again and again instead of being flagged. The code did what the tests asked for, and the tests checked the code. Nothing checked whether it did what we meant.

Neither the agent nor the reviewer was the cause. The cause was that nobody had written down what 'right' meant for that ticket. The intent existed in one engineer's head and in a Slack thread from months earlier. Agents optimise for whatever target they can see. If 'tests pass' is the only visible target, that's exactly what you get, with assumptions filling the gaps.

• Tests: does the code behave the way its author expected?
• Acceptance criteria: does the change do what the business and the user actually need?
• Evals: across realistic and adversarial cases, does the system keep doing the right thing over time?
• Stop conditions: when should the agent, or the person, stop trying and hand the problem back?

When agents write both the code and the tests, the first check more or less passes by default. If nobody owns the other three, a team can have high throughput and low correctness at the same time. The dashboard only shows the throughput.

3. The turn: I stopped writing code and took ownership of the eval suite

So I made a deliberate trade. I'd write very little production code and take ownership of the definition of done for the team's work. Day to day, that came down to three habits.

• Write acceptance criteria before the task starts. No ticket went to an agent or an engineer without a short list of checks someone could verify, written in plain language that also works as a prompt. 'Handle errors properly' doesn't count. 'Never retry on 4xx except 429; surface 4xx to the dead-letter queue within one attempt' does.
• Take eval cases from real incidents. Every postmortem, support escalation and 'how did this ship?' moment became a case in a shared eval set. Real failures beat made-up ones because they include the context nobody thought to write down.
• Make stop conditions explicit. Each task said when the agent should give up and hand back: unclear requirements, touching an auth or billing boundary, failing the same check twice, or needing a schema change. An agent that stops to ask costs far less than one that confidently makes something up.

Here's what one task spec looked like. It's short on purpose, because an imperfect spec is better than one nobody writes.

task: Add retry logic to partner webhook sender
owner: chi-cong

acceptance:
  - Retries only on 5xx, 429, and network timeouts
  - Max 5 attempts with exponential backoff, jitter, cap 60s
  - 4xx (except 429) goes to dead-letter queue after 1 attempt, with payload + response logged

evals:
  - id: incident-malformed-payload
    source: postmortem (webhook retry storm)
    given: partner returns 400 for a malformed body
    expect: exactly 1 attempt, DLQ entry created, no retry scheduled
  - id: partner-rate-limit
    given: partner returns 429 with Retry-After: 30
    expect: next attempt no earlier than 30s

stop_when:
  - Change requires modifying the shared HTTP client used by billing
  - Same acceptance check fails twice in a row
  - Partner contract is ambiguous about idempotency -> ask, don't guess

4. What changed next

I don't have a clean metric for you, and I'm not going to make one up. What changed was how the team behaved.

• Reviews got faster. Reviewers compared the diff with written criteria instead of reconstructing the intent in their heads. The 'is this what we wanted?' thinking had mostly been done before any code existed.
• Disagreements moved earlier. People argued about requirements ('should a 429 count toward max attempts?') rather than about style in diffs. That's a cheaper argument, and it happens before the work instead of after.
• Agents became more useful. Clear criteria and stop conditions meant agent output needed fewer rescue rounds, and when an agent did stop, it stopped somewhere sensible.
• My role shifted. People stopped asking me 'can you build this?' and started asking 'is this done?' I was relevant again because I owned the judgment, not the output.

5. The general principle

Whoever defines done sets the bar that every other contributor is measured against. Good tech leads have always worked this way. Their influence never came from how fast they typed. It came from knowing what 'finished' meant and holding everyone else to it. With agents, that becomes the whole job. When output costs almost nothing, the scarce thing is a clear definition of done that can actually be checked. You also can't hand it off. An agent can draft acceptance criteria, but it can't be accountable for whether they're the right ones.

When to use it, and when not to

• Use it for any task you give an agent, any cross-team change, anything that touches money, auth, data integrity or customers, and anything that has broken before.
• Use a lighter version for throwaway spikes, prototypes you'll delete, and exploration where figuring out what done means is the actual task. Write a stop condition (a timebox, or a question to answer) and skip the full criteria.
• Don't let it become ceremony. If writing the spec takes longer than reviewing the task, you've overdone it. Three checks, one eval and one stop condition is a starting size, not a minimum page count.

Common pitfalls

• Criteria nobody can check. Neither a person nor a machine can verify 'should be robust' or 'clean implementation'. Rewrite each criterion until someone could answer it with yes or no.
• Letting the agent write its own exam. If one agent writes the code, the tests and the acceptance criteria, it's grading itself. It can draft them if you want, but a human edits and owns them.
• Testing only the happy path. Your best eval cases come from incidents and edge cases. If none of your evals would have caught last quarter's bugs, you don't have evals yet.
• Leaving out stop conditions. Without them, agents keep going through ambiguity and produce confident nonsense. 'Stop and ask' has to be an allowed outcome, and one you reward.
• Letting the eval suite go stale. If incidents don't become new eval cases, the suite drifts away from the real system and the green checks start lying again.

6. Where this sits in the 30-day arc

Think back over the series. Ownership was about caring for outcomes and not just tasks. Communication was about making your thinking easy for others to follow. Visibility was about making sure people see your impact. Defining done uses all three. It's ownership written down, communicated in a form that both people and machines can check, and visible because everyone's work goes through it. If you've been practising the earlier lessons, you've been building this skill already.

7. The practice to start tomorrow

You don't need a new title or anyone's permission. You just need a ticket.

• Before your next ticket goes to an agent or a teammate, write three acceptance checks that can each be answered yes or no.
• Add one eval case based on a past bug in the same part of the system.
• Add one stop condition: the point where the work should pause and come back to a human.
• Put all of it in the ticket, the PR description, or the agent's task prompt (CLAUDE.md, AGENTS.md, or the prompt itself).
• Keep this up for two weeks, with a simple log: the ticket, which check caught something, and whether the existing tests would have caught it too.
• At the end, count how often your checks caught something the tests missed. That number is your evidence, for you and for your manager.

## Done means
- [ ] <check 1: observable behaviour, yes/no>
- [ ] <check 2>
- [ ] <check 3>

## Eval from past bug
- Source: <incident / ticket / postmortem>
- Given: <input or state>
- Expect: <exact outcome>

## Stop and hand back if
- <condition>

8. Close

This is the last lesson in the course, but you'll keep using the skill. Thirty days ago the question was how to grow as an engineer. For the next ten years, my answer is to become the person whose 'done' other people trust. Agents will keep getting better at producing output. A person still has to decide what that output is for and when it's good enough. Aim to be that person, and start with your next ticket.

Extend your knowledge

• Read Anthropic's engineering guidance on building effective agents and on writing CLAUDE.md files and task prompts. Notice how much of it is really about specifying done and saying when to stop.
• Read Hamel Husain on LLM evals, especially his advice to begin with real error analysis rather than generic metrics. It carries straight over to eval suites for agent work.
• Go back to Gojko Adzic's 'Specification by Example'. It predates AI, but its central idea, concrete examples as shared and executable definitions of done, fits agent task specs almost exactly.
• Take your team's last three postmortems and turn each one into a given/expect eval case. Put them in your next agent task prompt and watch what changes.