The PR Was 100% Correct — And It Still Took Down Three Services

Day 12 — Why the diff isn't the thing you're reviewing

You can approve an agent's PR, watch every line pass review clean, and still get paged two days later — because "is this code correct" was never the question that mattered. The question that matters is whether the diff matches the ask. That's a different check, and most of us aren't running it yet.

The PR that was "correct" and still took the system down

The ticket read: add a timeout to the payment webhook call so it doesn't hang the queue. The agent delivered exactly that — plus a refactor of the retry logic on the shared HTTP client, exponential backoff swapped in for the fixed delay, and a bumped default retry count. Every hunk reviewed clean. The timeout was correct. The backoff was textbook. The retry count was defensible. Nobody flagged a thing, because nothing was wrong with the code.

Two days later an unrelated downstream outage caused a spike in retries. The new backoff and retry defaults — now silently inherited by three other services sharing that client — doubled call volume into a dependency that was already struggling. No bug caused that incident. Code that worked exactly as written did it, doing something nobody scoped, load-tested, or told the other three call sites about.

The shift: from "is this code right" to "is this what was asked"

Human PR review grew up around a safe assumption: whoever wrote the PR scoped it themselves, against a task they own and have to finish cleanly. So review optimizes for correctness, readability, edge cases, style — because by the time it lands in front of you, scope is usually already sane.

That assumption doesn't hold for agents. An agent doesn't own a task the way an engineer owns a sprint — it owns "be maximally helpful on this prompt, right now." So the review question has to move one level up, before correctness: does this diff's scope match the ask, full stop. If it doesn't, the correctness of the extra code is beside the point. It shouldn't get evaluated as good code — it should get evaluated as an unscoped change that happens to compile.

Why agents overreach — it's structural, not sloppiness

This isn't an agent having a bad day. It's the predictable output of how these systems are built and used:

• ▹Agents are optimized to be thorough and helpful on the visible task — "while I was here, I also cleaned up the retry logic" reads as diligence to the training signal, not as a scope violation.
• ▹There's no cost function for stopping. A human engineer feels the social cost of a drive-by refactor breaking someone else's service; an agent carries none of that cost forward into the next PR.
• ▹Agents pattern-match "good engineering practice" — dedupe this, add error handling there, rename for consistency — and apply it wherever the file happens to already be open, because in isolation it looks like quality.
• ▹The harness rewards finishing the task, not holding its boundary. Nothing in a typical agent loop scores "correctly declined to touch the other three call sites."

The mechanic: diff the diff against the prompt, not the code against itself

Before you open the diff, write down the smallest correct scope in one line — the actual ticket text, or the prompt you gave the agent. Then read the diff hunk by hunk and ask, for each one: does this trace back to a word in that line? If yes, review it normally for correctness. If no, it doesn't get auto-rejected — but it gets pulled out and justified on its own, explicitly, not smuggled in because the file was already open.

Ticket: "Add pagination to GET /users (limit/offset, default limit=25)"

Diff includes:
  + users_controller.rb   (pagination added)         -> maps to ticket
  + user_serializer.rb    (default page_size changed) -> maps to ticket
  ~ base_serializer.rb    (default page_size changed for ALL resources) -> NOT in ticket
  ~ orders_controller.rb  (inherits new default)       -> NOT in ticket

The first two hunks pass correctness review easily.
The last two are the ones that cause the incident three sprints from now.

A second example from PhoenixDX review practice

Ticket: add pagination to the users list endpoint. The agent's PR added pagination correctly. It also changed the default page size on the shared base serializer "for consistency," which silently changed response shape on two other endpoints. The pagination logic was fine. The serializer change was fine in isolation. It only got caught because a reviewer went back to the ticket and noticed it named exactly one endpoint. The bug wasn't in the diff — it was in the gap between the diff and the ticket, and that gap is invisible if you only ever read the code.

Turn it into a habit

One question, asked before you assess correctness, on every agent PR: what's in here that nobody asked for?

• ▹If the answer is nothing — proceed to normal review, correctness and style, as usual.
• ▹If there's something — it doesn't automatically get rejected, but it gets its own justification, its own blast-radius check, and ideally its own PR or ticket.
• ▹Say it out loud in review comments. "This hunk isn't in the ticket, why is it here" is a completely legitimate review comment — make it a normal one.

Closing this Day 12

This is Day 12 of Software Engineering in the AI Era. Tomorrow: what happens when scope creep passes review anyway — because this discipline catches most of it, not all of it, and the failure mode when it slips through looks nothing like a normal bug escape.

Extend your knowledge

• ▹Pull up your team's PR template and check whether it asks 'what does this PR explicitly NOT do' — if not, that's a one-line addition worth making for agent PRs specifically.
• ▹If you're using an agent harness like Claude Code or Cursor, try requiring the originating ticket/prompt link in the PR description by default, so the mapping step has a fixed reference point instead of relying on reviewer memory.
• ▹Revisit any internal 'scope discipline' norms your team already has from the pre-agent era (bounded-diff PR culture, single-responsibility commits) and ask which of them need to be made explicit now that the author isn't a person with skin in the game.
• ▹Tomorrow's Day 13 piece: what happens when scope creep passes review anyway.