"What's the failing seed?" That was the whole review comment. It landed on the fourth concurrency PR our coding agent had opened that week. The first three were already merged. Across the four we had roughly 400 lines of locking, retrying and pool refactoring, and every one of them shipped with a green build. Nobody replied, because nobody had a seed. Nobody had a reproduction of any kind. The agent had fixed the same race condition three times, with a passing build each time, and the race condition never existed.
I keep coming back to one question. If those green builds didn't prove the fixes, what did they prove?
Day 1: a good hypothesis
Monday. One integration test goes red on main: TestPoolCheckoutUnderLoad, in the service that owns our database connection pool. Our test harness has a watchdog that dumps every goroutine stack when a test blows its deadline, so the failure log arrived with a wall of stack traces. Two goroutines were parked in pool.acquire. A third was inside pool.release.
The agent picked up the ticket and, before touching any code, wrote this in its working notes:
Observed: test exceeded deadline. Goroutines 41 and 57 both blocked in
pool.acquire(); goroutine 62 in pool.release() holding p.idle.
Hypothesis: release() appends to p.idle and signals p.cond without holding
p.mu for the full critical section. If acquire() checks len(p.idle) between
the append and the signal, it waits on a signal that has already fired.
That's a lost wakeup, which would explain a hang that only shows up under load.
Plan: hold p.mu across append+signal in release(). Rerun the test to confirm.Read it again. That's good reasoning. It names a specific mechanism, ties it to the evidence, and proposes the smallest change that would test it. If a mid-level engineer on my team wrote that, I'd be pleased. And that's what makes this uncomfortable. The agent's thinking was fine. What it was thinking about was the problem.
Days 2 to 4: the loop
Here's how the week actually went, with the dice rolls shown. We only learned this later, from the history: the test passes about 86% of the time, roughly 6 runs in 7, whatever the code does. Hold onto that number while you read.
- Day 1, afternoon. The agent wraps append+signal in the mutex and reruns. Green (an 86% roll). PR #1 goes up as "fixes lost wakeup in pool.release". Sensible review, green CI, merged.
- Day 2. The test fails again, this time on someone else's unrelated PR. The agent reads the new dump and decides fix one was necessary but not sufficient, because acquire() can also race with the idle reaper. So it adds a second lock for the reaper, plus a retry wrapper around checkout "to tolerate transient contention". Green (86% roll). PR #2, merged.
- Day 3. Red again. Now the agent's view is that three locks make the whole locking model fragile, and it refactors the pool to a single channel-based design. About 250 lines. Green (86% roll). PR #3, merged.
- Day 4. Red again, on the new design. The agent finds a 'remaining edge case' in channel close ordering and patches it. Green. PR #4, and that's where the reviewer asked for the seed.
The arithmetic doesn't care how good the reasoning was. If a test passes 6 times in 7, you'll get a green rerun after any change about 86% of the time. Three fixes in a row, each followed by a green rerun, happens about 63% of the time with zero real fixes. So 'make a change, rerun once, see green' most likely ends exactly where we ended up. Each 'fix' was a coin flip that happened to land the right way, and each lucky flip got written up as confirmation.
There was a clue sitting there the whole time. CI runs with -race, and Go's race detector didn't report a single data race in any of those runs. Nobody weighed that, the agent included. A clean race detector doesn't prove there's no race. Still, four days of silence from it should have counted for something.
Why four reviews didn't catch it
Read on its own, each PR made sense. The descriptions were fluent and specific. They quoted line numbers from the goroutine dump. CI was green. Reviewers did what reviewers do: checked that the explanation held together and that the diff matched it. Every time, it did.
That's the trap. Our reviewers were checking the reasoning, and reasoning is the cheapest thing an agent makes. Hand it any stack trace and it'll give you a convincing causal story in seconds, as many times as you like. A reproduction costs far more to produce, and it was the one thing missing from all four PRs.
The 90-day history
After the seed comment, one of our engineers pulled 90 days of CI results for the test. The failure rate was flat at roughly 1 in 7, and it had been since long before any agent touched the repo. The real cause was boring. The test asserts a checkout deadline that a shared CI runner under load sometimes can't hit. The timing assumption in the test was wrong. The pool code was fine.
When we raised it at standup, two people said nearly the same thing: "Oh, that one. Just rerun it." They'd known for months and had rerun it dozens of times on autopilot. What they knew was real and it was correct, but it lived only in their heads, and the agent had no way to reach it.
The reframe
The agent wasn't wrong. It acted like a careful engineer seeing this test fail for the first time: take the red build at face value, form a hypothesis, change the code, check the result. That's the right method. The trouble was the combination: no memory of past flakes, plus persistence, plus speed. A human would have stopped by day two. Some of that is fatigue, but mostly it's a little voice in the back of your head saying "that test always does that". The agent starts every session without that voice.
To a human, a flaky test is an annoyance. To an agent, it's a lying oracle. Inside a session, the test result is the closest thing the agent has to a definition of 'done', so a green rerun works as a reward. Make the reward noisy and the agent will fit a story to the noise, and every lucky roll makes the story stronger. ML people have a name for optimising against a broken score instead of the real goal. You don't need the name to spot it in your own repo. The agent did what we rewarded it for, and we were rewarding coin flips.
So a smarter model wasn't the fix. A smarter model would just have told a better story. The agent needed a test signal that told the truth.
What we built, part 1: write the folklore down
We added a machine-readable reliability file to the repo, regenerated every night from CI history. Each test gets a tier (trusted, probation or quarantined), a rolling pass rate over its last 200 runs, and a flake budget. Fail more than 2% of the last 200 runs and the test drops a tier. To climb back up, it has to stay inside the budget for a full window.
# .ci/test-reliability.yaml
# Generated nightly from CI history. Edit tier/notes by PR; never hand-edit stats.
policy:
window_runs: 200
flake_budget: 0.02 # >2% failures in window => drop one tier
promote_after_window: true # must stay in budget for a full window to move up
default_tier: probation # tests not listed here
tests:
- id: services/pool/integration::TestPoolCheckoutUnderLoad
tier: quarantined
runs: 200
failures: 29
pass_rate: 0.855
first_flaky: 2026-05-14
owner: platform
ticket: PLAT-1182
note: Checkout deadline too tight for shared runners. Not a known pool defect.
- id: services/billing/integration::TestInvoiceRetryIdempotent
tier: probation
runs: 200
failures: 6
pass_rate: 0.970
owner: billing
- id: services/pool/unit::TestAcquireReleaseOrdering
tier: trusted
runs: 200
failures: 0
pass_rate: 1.000
owner: platformOn its own the file does nothing. It only works because the agent is told what each tier means. These are the lines from our agent instructions file, and they're short on purpose:
## Test failures
Before reasoning about any failing test, look it up in .ci/test-reliability.yaml.
- trusted: a failure is real evidence. Investigate it.
- probation (or not listed): run the retry protocol below before forming a hypothesis.
- quarantined: a failure is NOT evidence. Never change production code to make it pass.
If your change touches what it covers, write a deterministic test instead.
A single green rerun never confirms a fix. Say what reproduction you have, or say you have none.What we built, part 2: retry before you reason
The file says how far to trust a test. The protocol covers what to do when that's unclear. Before writing any hypothesis about a probation or unlisted test, the agent reruns that single test in isolation 20 times, on its branch and on the merge base, with caching off:
# scripts/rerun-isolated.sh <package> <TestName> <ref>
git worktree add -f /tmp/rerun-$3 "$3" >/dev/null
cd /tmp/rerun-$3
go test -run "^$2\$" -count=20 -race -shuffle=on "$1" 2>&1 \
| tee /tmp/rerun-$3.log \
| grep -E '^(--- FAIL|ok|FAIL|-test.shuffle)'The merge-base comparison is what keeps this honest. You stop hunting for one green run and start measuring a rate against a baseline. Twenty runs of a 1-in-7 test will usually give you two or three failures on main and two or three on the branch. Same rate on both sides means the failure has nothing to do with your change. If it reproduces deterministically, or shows up clearly more often on the branch than on main, it's probably yours. Go investigate.
The escape hatch mattered more than the file did. If the failure won't reproduce in isolated reruns and the test is over budget, the agent files a quarantine ticket instead of a code change: test ID, both rerun logs, the shuffle seeds, the historical rate. Then it stops.
Here's why that's the important bit. Looking back at the week, the agent had exactly two moves: open a PR, or keep digging. Nothing in its instructions or our workflow treated 'this red build isn't mine' as a valid result, so every ambiguous signal became a diff. If you want an agent to stop fixing ghosts, 'not my bug' has to be an outcome it can actually reach, write up and get credit for. It should count as a finished task, same as a merged PR.
What we built, part 3: the review rule
The postmortem gave reviewers one rule, and it's the one I'd steal first: no deterministic reproduction, no concurrency PR. 'Deterministic' means one of three things:
- A seeded test that fails on the base commit and passes on the branch, with the seed in the PR description. The seed has to control whatever drives the bad ordering. Careful here: the seed go test -shuffle prints only fixes test order, not goroutine scheduling.
- A forced interleaving. Test hooks or channels push the goroutines into the bad ordering every time, so the test fails 100 out of 100 runs before the fix.
- A recorded trace from a real failure, either a race detector report or an execution trace, showing the conflicting accesses.
Reviewers check that the repro exists and that it fails without the change. They don't grade the argument. That was the point of the original question. 'What's the failing seed?' was the right thing to ask all along. We just turned it from a lucky moment in review into a requirement.
Aftermath
We reverted the three merged PRs and closed the fourth, roughly 400 lines in total. The retry wrapper from PR #2 scared me most. It would have hidden real checkout failures from callers in production, and nobody noticed because it looked like defensive programming. We loosened the test's deadline and quarantined it with a ticket. A few days later the platform team rewrote it on a fake clock, and it's now working its way back to trusted.
The first nightly run of the reliability job quarantined 23 integration tests. We'd asked the team to guess beforehand, and people said three or four. For most of those 23, someone on the team 'just knew'. A few had nobody, and those are the ones that worried me.
About a week later the agent hit a red build on a billing test that was on probation. It ran the reruns: three failures in 20 on the branch, three in 20 on main. It filed a quarantine ticket with both logs attached and didn't open a PR. The ticket ran four paragraphs and was better documented than most flaky-test tickets I've written myself. For me, that was when the incident actually ended.
The question to take back to your repo
Once agents are reading your CI, test reliability data stops being a dashboard humans glance at when they're curious. It becomes part of the context the agent works from, and the agent will act on it completely and without suspicion. So it has to be accurate, it has to live in the repo, and it has to come with instructions for what to do when the signal is bad.
Which of your tests does everyone 'just know' about? And where is that written down?