Every Commit Was Correct. The Runbook Still Nearly Killed a Failover.

3am. Database failover. The on-call engineer is doing everything by the book — literally. Step 6: drain and restart the connection pooler before promoting the replica. He does it. Twelve minutes later, step 9: do not touch the pooler until after promotion completes and DNS has propagated. He already broke that rule three steps ago. Now he's staring at a wall of dropped connections that neither instruction, read alone, could have warned him about.

Here's the thing — neither step is wrong. Step 6 exists because in an earlier incident, app servers kept writing to a dead primary through stale pooler connections for four straight minutes; restarting the pooler first was the fix that stopped the bleeding. Step 9 exists because in a more recent incident, someone restarted the pooler too early and threw away perfectly good in-flight reads the still-healthy primary could have served during the promotion window. Alone, each is sound engineering judgment. Back to back, step 6 orders you to do the exact thing step 9 tells you not to do yet.

The git blame

Mid-postmortem, we pulled up the history on both steps, half-expecting to find some sloppy copy-paste or an ignored review comment. Instead: two clean, well-reasoned PRs. Step 6 landed three months ago — one line added to an existing bullet, opened right after Incident A's postmortem, approved in eight minutes by a reviewer who'd been paged into that same incident. Step 9 landed six days before our 3am call — a new bullet inserted three steps later, opened right after Incident B's postmortem, approved in six minutes by a different reviewer who hadn't been anywhere near Incident A.

commit 8f2c1a0  (3 months ago)  author: priya
    runbook: restart pooler before replica promotion
    Fixes stale-connection cascade from INC-1042. Reviewed 8m, approved.

commit e91d47b  (6 days ago)  author: marco
    runbook: don't restart pooler until after promotion + DNS propagation
    Prevents dropped in-flight reads seen in INC-1198. Reviewed 6m, approved.

Both reviewers did their job. Both diffs were true statements about the incident that prompted them. Neither reviewer had the context — or honestly, the reason — to reread the other eleven steps in the document and ask whether this new sentence quietly reversed an instruction two steps up. Why would they? The PR was three lines.

Nobody can name the bad commit

That line stuck in the postmortem: there is no bad commit. We went looking for it anyway, the way you go looking for the line that broke the build — we wanted a villain, someone who skipped a test, skipped a read, skipped a thought. Instead, every diff in that document's history, read on its own, was correct for the incident that produced it. The runbook didn't get corrupted by one bad edit. It decayed through roughly a hundred individually correct ones, each patching the step in front of it without rereading the whole document, until the thing converged on something locally coherent and globally broken.

Diffs are the wrong unit of truth for a living procedure

Here's the mechanism, stated plainly: code review answers "is this patch correct," not "is the document still correct." For most code, that gap doesn't matter much — tests and types catch the cross-cutting breakage a single diff can't see. A runbook has no compiler. It's prose, read top to bottom under pressure, and its correctness lives in the relationship between steps, not inside any one of them. A review process built around the diff — human reviewer or agent, doesn't matter which — is structurally blind to a failure mode that only exists in the gap between two diffs that never shared a PR. This is exactly what surfaces the moment you hand agents write access to any shared document with a long edit history and no single owner rereading it end to end: every edit is locally rational, and the aggregate drifts somewhere nobody chose.

Grep for more of these

Once we had a name for the failure mode, we stopped treating step 6/step 9 as a one-off and went hunting for its siblings. The method is boring, and it works: walk the runbook's git history, pull every added line, cluster by the system resource it touches — pooler, cache, DNS, feature flags, replica state. Anywhere the same resource gets touched by edits from two different incidents, read both edits in the order they'd actually execute, not the order they were written.

# list every added line across the runbook's history, oldest first,
# so you can spot two edits about the same resource written months apart
git log --follow --diff-filter=AM --reverse -p -- RUNBOOK.md \
  | grep -E '^\+' \
  | grep -Ei 'pooler|pgbouncer|redis|cache|replica|dns|feature.?flag' \
  | nl

Took ten minutes to find a second one. Step 14, added five months ago after a caching incident, says to manually warm the Redis cache before marking the service healthy again. Step 21, added six weeks ago after an unrelated incident where a stale cache was masking a config bug, says to flush Redis first thing when declaring any incident, to rule out stale-cache causes as a first move. Run both, in order, on an incident that turns out to be a false alarm halfway through: you flush a cache you didn't need to flush, then warm it back up, and you've manufactured a cold-cache latency spike out of a non-event. Same shape as the pooler contradiction. Different resource, different incidents, same blind spot.

What we actually changed

• ▹Not "review agent-written runbook edits more carefully." That's the same diff-by-diff blind spot, just slower and dressed up with more false confidence.
• ▹A standing quarterly reread of the whole document, owned by one named human per runbook. The only deliverable: does this still read as one coherent procedure — not a line audit, a cover-to-cover read.
• ▹Every step now carries a tag for the system state or resource it touches (pooler, cache, DNS, flags...) in a comment header, so a reviewer — or an agent — can ask "what else touches this resource" before approving a new step nearby.
• ▹Any new step touching a resource another step already touches gets routed to that runbook's human owner, no matter how small the diff looks.

None of this fixes the underlying question — it just buys us a shot at catching the drift before an incident does instead of during one. The harder question I don't have a clean answer for, and it's worth sitting with if you're doing any multi-agent work: how do you even define "wrong" for a document where every sentence is true and the document as a whole is false? That's not a runbook problem specifically. It's what's waiting at the end of any process where agents get write access to a shared artifact that nobody — human or agent — ever rereads end to end, only diffs. The runbook was just the first place it landed on my desk.