Three Coding Agents Each Fixed the Same Bug Correctly. Together They Broke It Worse.

Three engineers 'fixed' the same flaky test in the same week. Three PRs, three green CI runs, three approvals. Two weeks later it flared up again — worse — and git blame lined up three coding agents, none of which had any idea the other two existed.

The week three fixes shipped

The test was test_webhook_delivery_confirms — flaky enough that everyone on the notifications squad had a personal theory about it, never flaky enough to block a release. The kind of test you learn to live with. Second week of August it flaked three times in three different PRs, and three engineers, each with an agent open, each under mild deadline pressure, independently decided this was finally the week to kill it.

• ▹Monday, Aug 11 — PR #1842 (Minh): wraps the delivery check in a retry loop. 'Test was racing the dispatcher, retry absorbs the race.' CI green.
• ▹Wednesday, Aug 13 — PR #1858 (Sarah): mocks the clock so the timeout can't fire early. 'Test was time-dependent, freeze time.' CI green.
• ▹Friday, Aug 15 — PR #1871 (Diego): widens the assertion window from 200ms to 800ms. 'Test was too strict for CI's noisy neighbor load.' CI green.

Three green merges, three approved reviews, and in none of the three PR threads did anyone ask whether the other two existed — because at merge time they didn't yet, or nothing pointed the reviewer toward looking.

The postmortem

On Aug 29 the test flaked again, and this time it took a release candidate down with it. Run git log --follow on the test file and all three commits line up inside four days — and the diffs explain exactly why none of the fixes held. Minh's retry loop was quietly masking a real concurrency bug in the dispatcher. Diego's widened assertion window hid the exact symptom the retry loop was supposed to catch, because now a delivery that arrived dangerously late just... passed. And Sarah's mocked clock turned out to be solving a problem that never existed — the test was never actually time-dependent. Each fix was locally correct against the failure its author was staring at. Stacked together, they built a test that could no longer fail even when the dispatcher was genuinely broken.

Same bug, three transcripts

I pulled the three agent transcripts side by side — Minh's, Sarah's, Diego's — expecting to find someone being careless. Instead I found three self-consistent, well-reasoned diagnoses, each one perfectly explainable by what got pasted into that session:

• ▹Minh's session started from a CI log showing the assertion firing 40ms after a retry-eligible dispatch — reasonable conclusion: add retry.
• ▹Sarah's session started from a Slack thread three weeks old, someone speculating the test 'seems time-sensitive' — reasonable conclusion: freeze the clock.
• ▹Diego's session started from a different CI run entirely, one where CI itself was under load and the timing was genuinely tight — reasonable conclusion: the threshold is too strict.

Every agent reasoned correctly. Every agent was working from a true but tiny sample of the same underlying population of failures, and had no way of knowing it was a sample rather than the whole story.

This wasn't carelessness — it's coverage

That's the actual failure mode, and it's worth naming precisely because it's not 'engineers should be more careful' or 'agents hallucinate root causes.' No single context window, human or agent, ever held the full failure history of this test. Each session got a slice — one CI run, one Slack thread, one flaky-test ticket — and produced a diagnosis that was optimal given that slice. The bug wasn't in any individual's reasoning. It was in the fact that three non-overlapping slices of the same population produced three contradictory theories, and nothing in the process ever compared them.

Why review didn't catch it

This is the part that stings, because code review did its job. Each diff was small, each rationale was plausible, each PR read cleanly against the file as it stood. Reviewers approved a diagnosis-shaped diff, not a diagnosis — nobody's job in that process was to ask 'does this contradict a fix from two days ago.' That comparison requires holding all three contexts in your head simultaneously, and no human reviewer and no agent session was ever positioned to do that. The review process was scoped to the diff. The bug lived in the relationship between three diffs that never appeared in the same room.

The reframe

We'd been treating context window management as a per-engineer prompting skill — what do I paste into the session today, how do I phrase the task. That framing is wrong at the team level. The actual gap is infrastructure: is there a shared, durable record of 'why is this test flaky' that any agent, on any given Tuesday, can be pointed at before it touches the file? Individually skilled prompting doesn't fix a problem that only exists at the level of the team's collective memory.

What we changed at PhoenixDX

Nothing fancy — no new tool, no dashboard. Just a convention: every flaky test gets a running log, checked into the repo next to the test file, that every agent session is required to read before it's allowed to propose a fix. Not a ticket that lives in a separate system nobody opens mid-session — a file the agent's own context-gathering step would naturally pick up.

# FLAKY.md — test_webhook_delivery_confirms

## 2025-08-11 (Minh, PR #1842)
Hypothesis: dispatcher races the delivery check.
Fix: retry loop around the assertion.
Status: did NOT fix root cause — see 2025-08-29.

## 2025-08-13 (Sarah, PR #1858)
Hypothesis: test is time-dependent, real clock drifts under CI load.
Fix: freeze clock with mock.
Status: unverified against Minh's retry fix — never compared.

## 2025-08-15 (Diego, PR #1871)
Hypothesis: 200ms window too strict for noisy CI.
Fix: widened to 800ms.
Status: masked the regression retry was meant to catch.

## 2025-08-29 (Anh, PR #1904)
Read all three entries above before touching this file.
Root cause: dispatcher genuinely deadlocks under concurrent retries — Minh's
retry loop and Diego's widened window combined to hide this. Sarah's mock
clock was irrelevant to the real bug all along.
Fix: dispatcher mutex + revert #1871's window widening.

When Anh picked up the flake on Aug 29 — the fourth fix in three weeks — the difference wasn't a smarter agent. It was that her session started from the accumulated diagnosis instead of one CI screenshot. She could see that Minh's and Diego's fixes contradicted each other before writing a line of code, which is the comparison none of the previous three sessions were ever positioned to make.

The uncomfortable generalization

This isn't really a story about flaky tests. It's what happens to any shared codebase once agents make individual iteration cheap enough that nobody has to talk to anyone before touching the same file. When a fix took a human half a day, you naturally bumped into your teammates' work-in-progress — standups, Slack, the PR sitting open for review. When a fix takes an agent twelve minutes, three people can independently 'solve' the same problem in the same week and never once collide in a way that surfaces the contradiction. The bottleneck didn't disappear when agents got fast. It moved. It used to be writing the fix. Now it's knowing someone already tried.