Reversal criteria

Day 29: Make the Change Falsifiable — Reversal Criteria Beat Vision Decks

Your loudest skeptic doesn't care about your vision. They want to know what result would make you stop. Tell them before launch, write it down, and the person most likely to sink your rollout can end up running the scoreboard for it. You've been asked to change how your team works, and the people you most need are the ones who trust slides least. Engineers don't believe a change because it comes with a good story. They believe it when you've said in advance what would make you drop it.

The hook: one sentence did what the deck couldn't

When we brought AI coding agents into our delivery workflow, I did what most managers do and showed a vision deck. Faster cycle time, less toil, engineers moving up into design and review. Nobody moved. The polite people nodded. Our loudest skeptic was a senior engineer who had watched plenty of tooling fads come and go, and he asked one question: "And when it doesn't work?"

I didn't have a good answer that day. The next day I posted one sentence in the team channel: "If review time per PR goes up for three weeks in a row, or escaped defects pass our threshold, we roll the agent workflow back at the week-6 review, and you own the numbers." Two weeks later he was posting the weekly numbers himself, and nobody had asked him to. So the deck failed and the reversal sentence worked.

Why selling the 'why' falls flat with engineers

The standard change-management advice is to start with the why. That works on people who haven't been burned yet, and most engineers have. They've lived through the reorg that was going to fix everything, the platform migration that stalled at 60%, and the 'we're all-in on microservices' memo that nobody brought up again. Big announcements usually fade out without a post-mortem.

• ▹A skeptic isn't asking 'why are we doing this?' They're asking 'what happens if this doesn't work?'
• ▹A vision can't answer that, because it only describes the upside.
• ▹If nobody answers, the safe play is to resist early. Once the change ships, nobody will ever officially say it failed.
• ▹Reversal criteria answer the question head-on. This is what failure looks like, this is when we check, and this is what we'll do.

AI makes this sharper. Every vendor demo of a coding agent looks great, and your engineers know a demo isn't their codebase. Their scepticism is a rational prior, not a character flaw, and reversal criteria treat it that way.

The research idea behind it: unfalsifiable claims aren't worth trusting

Karl Popper's point is simple. If no observation could ever contradict a claim, the claim isn't strong. It's empty. 'AI agents will make us more productive' is unfalsifiable as stated, because any bad week can be explained away. In my PhD work on multi-agent systems, we don't accept a change to an agent workflow (a new planner, a new tool, a new handoff protocol) until the failure condition is written down before the run: which metric, which threshold, which benchmark. Skip that step and you end up telling a story about the results after you've already seen them. I brought that habit into management almost unchanged. Changing a team's process is an experiment on a socio-technical system, and it deserves the same discipline.

The move: three parts of a reversal criterion

Write the criteria before launch and publish them. Then repeat them more often than you repeat the vision. Every criterion has three parts.

• ▹1. The metric and the threshold. It has to be specific, measurable, and something the team already believes in. For example: 'escaped defects per sprint rise above our trailing 3-sprint average + 1', 'median review time per PR goes up for 3 consecutive weeks', 'percentage of agent-authored PRs reverted within 7 days exceeds 10%', or 'on-call pages traced to agent-written code exceed N in the trial period'. Take the thresholds from your own baseline. Don't take them from a vendor's marketing.
• ▹2. The decision date. It's a fixed day on the calendar, e.g. 'week 6 review, Thursday'. Without one, the trial quietly becomes the new normal, which is exactly the slow fade skeptics are afraid of.
• ▹3. The evidence owner. Name one person, and make it your biggest skeptic. They pull the numbers, they post them, and they can call a criterion tripped. Their scepticism stops being resistance and becomes your measurement system.

Template: the 5-line reversal contract

Post this in the team channel. A doc nobody opens won't do the job. Below is the version from our team's agent rollout. Swap in your own baselines.

CHANGE: Coding agents draft PRs for tickets labelled `agent-ok`; humans review everything before merge.
WE ROLL BACK IF: median review time per PR rises 3 weeks in a row, OR escaped defects per sprint exceed our 3-sprint baseline + 1, OR >10% of agent PRs are reverted within 7 days.
WE DECIDE ON: Week 6 review (Thu, 26 Nov), 30 min, whole team.
EVIDENCE OWNER: <biggest skeptic> pulls the numbers every Friday and posts them here, unedited. They can call a trip at any time.
I COMMIT TO: not changing these criteria mid-trial. If I think one is wrong, I'll say so publicly and the change starts at the next trial.

The last line matters most. It's your promise not to move the goalposts, and the team will watch whether you keep it.

What changed on our team

• ▹Arguments stopped being opinions and turned into data. 'Agents write junk' became 'review time went up this week, mostly in the payments service'. We could act on the second version, for example by taking that service off the `agent-ok` label.
• ▹Problems came up earlier. Before the contract, calling out an agent failure felt like being disloyal to the initiative. After it, you were just filling in the evidence. People flagged bad agent PRs in week one, not week four.
• ▹The skeptic became the most trusted voice on the results. When he said a number looked fine, the team believed him in a way they never would have believed me.
• ▹The week-6 decision wasn't a vote on me. We checked the numbers against criteria everyone had agreed to six weeks earlier, kept the workflow, and narrowed the scope where review cost had gone up. Nobody had to win, and nobody lost face.

How this goes wrong

• ▹The criteria are too loose to ever trip. 'We'll roll back if quality drops significantly' isn't a criterion. If you can't name the exact number that trips it, rewrite it. Ask yourself whether it could realistically trip in a bad trial. If it couldn't, it's theatre, and engineers will see through it.
• ▹You quietly move the goalposts. A number looks bad, so you redefine 'escaped defect' or restart the baseline from a better sprint. This is worse than having no criteria, because it proves the contract was never meant to bind you. If a criterion really was wrong, say so in public and fix it for the next trial, not this one.
• ▹You pick metrics the agents can game. PR count, lines changed and tickets closed all go up as soon as an agent is involved, so they tell you nothing. As the metrics lesson earlier in the series covered, choose outcome and cost metrics (defects, reverts, review burden, incidents) over activity metrics. Goodhart's law hits harder when the activity comes from a machine that never gets tired.
• ▹You set only one criterion. You can improve one metric by making another worse, so pair a speed metric with a quality metric. Then neither can be gamed on its own.
• ▹The evidence owner is on your side. If the owner is your ally, the skeptics won't trust the numbers, and you're back to arguing about credibility.

The tough case: the criteria trip, but the decision isn't yours

A lot of AI rollouts are mandates from above. Your criteria trip and you don't have the authority to roll back. You still owe the team three things:

• ▹Publish the result to the team, exactly as you promised. 'Criterion 2 tripped. I don't own the rollback decision, and here's what I'm doing about it.' If you hide the result, you throw away all the trust the contract earned.
• ▹Escalate with evidence, not a complaint. Bring the pre-registered criteria, the baseline, the weekly series and a specific proposal: narrow the scope, change the review policy, pause on certain services, or extend the trial with a fix. Leadership can still override the data, but with a written record in front of them they'll have to do it on purpose.
• ▹Tell the team what was decided, and if leadership overrode you, tell them why. Being honest in both directions is part of the job. It isn't a political risk to manage. Managers who agree in public and then say nothing about the result lose their team without noticing.

Google's SRE error budget is a useful comparison. The SLO and the policy for what happens when the budget runs out are agreed before any incident, so the argument about what to do happens while everyone is calm, not in the middle of a crisis. A reversal contract is an error budget for a process change.

How this fits the series

Earlier days covered building trust (say what you'll do, then do it) and measuring (outcome metrics over activity metrics, especially when agents produce the activity). Day 29 applies both to change. A reversal contract is a measurable promise that you make in public and then keep. Day 30 pulls the whole series into one operating system for leading an AI-era team.

Your one action this week

Pick the change you're pushing right now. It might be the agent rollout, a new review policy or a new on-call rotation. Write its reversal criteria in five lines or fewer: metric and threshold, decision date, evidence owner. Then send it straight to the person most likely to say it won't work, and ask them to own the evidence. If you can't write the criteria at all, take that as a warning about the change.

Extend your knowledge

• ▹Start with the 'Embracing Risk' chapter in Google's 'Site Reliability Engineering' book, then the example error budget policy in its companion, 'The Site Reliability Workbook'. Both are free online, and they're the closest engineering analogue to a reversal contract.
• ▹Gary Klein's 'pre-mortem' method (HBR, 'Performing a Project Premortem') is worth reading. Run one with your team before launch to come up with candidate reversal criteria.
• ▹Annie Duke's 'Quit' covers setting kill criteria in advance, and explains why people resist quitting even when the evidence tells them to.
• ▹Go back to 'Accelerate' (Forsgren, Humble, Kim) and the DORA metrics when you pick outcome metrics for your agent rollout. They're hard to game.