Flaky tests and coding agents

I approved a PR whose summary said: "test_auth_refresh: known intermittent failure, passed on rerun." Two days later staging was rejecting every refresh token issued after a key rotation, and users were getting logged out mid-session.

Then I pulled up the test's history. test_auth_refresh had never flaked. Not once. It had been green on every main-branch run since the day it was written. The agent had rerun a real regression four times, got one lucky green, and described it in words it had picked up from a different test 40 turns earlier in the same session.

So where did it learn that red means retry?

Rewinding the transcript

Turn 12: test_payment_webhook times out. That one really is flaky. It talks to a local stub that sometimes starts too slowly under parallel workers, and the team had said as much in chat that morning. The agent reruns it, it passes, and the agent notes: "flaky, known issue." Correct call. Any of us would have done the same.

Turn 52: test_auth_refresh fails with an AssertionError. Not a timeout. Different file, different subsystem. The agent reruns it. Red. Again. Red. Again. Red. The fourth run passes, because the bug only showed up depending on which side of a second boundary the rotated key's timestamp landed. The agent writes "known intermittent failure, passed on rerun", lifting the wording almost word for word from turn 12.

I'd assumed the agent took away "payment_webhook is flaky." The transcript said otherwise. What it actually learned was "when a test goes red, rerun until it goes green, then call it known."

Flakiness spreads through the context window

Everyone on your team carries a private list of which tests flake and how. When payment_webhook times out, you shrug. When auth_refresh throws an assertion error, you sit up, because you know that test doesn't do that. The knowledge is tied to one test and one kind of failure.

An agent has no such list. It has a context window full of examples, and it learns from examples, so a single "rerun fixed it" reads like a demonstration of policy. Nothing in the transcript pinned the turn 12 lesson to payment_webhook. The only anchor was a test name, and test names are easy to skip past when a broader pattern (rerun, see green, move on) also fits.

Compaction makes it worse. When a long session gets summarized, "ran the suite, tests pass" survives and "three failures before the pass" doesn't. The summary keeps the conclusion and throws away the evidence, so later turns inherit the habit without the detail that should have made it look suspicious.

Our first fix made it worse

The obvious move was a quarantine list in CLAUDE.md, so the agent would know which tests were actually flaky. It looked like this:

## Known flaky tests (quarantined)
These tests are known to be intermittent. If they fail, rerun once.
If they pass on rerun, note it in the PR and continue.

- test_payment_webhook
- checkout_test
- test_inventory_sync
- test_email_digest_schedule

A week later an agent opened a PR that touched the checkout flow. checkout_test failed. It was on the list, so the agent reran it, got green on the second try, and wrote "checkout_test: quarantined flake, passed on rerun."

checkout_test's known flake is a timeout waiting for a mocked payment iframe. This failure was an AttributeError ('NoneType' object has no attribute...) in the discount calculation. The agent had introduced it, and it only fired when the cart held a gift card. The rerun passed because the fixture picks cart contents from a random seed.

The list matched on test name and ignored the failure itself. In effect it handed a standing pardon to any bug that could reach a quarantined test. checkout_test exercises half the checkout flow. That's a lot of pardoned bugs.

Fix 1: fingerprint the flake, not the test

A flake has a shape: an exception type, a message, the frame where it blows up, and the conditions it needs. Anything outside that shape is a regression, even inside a quarantined test. This is the block we keep in CLAUDE.md now:

## Flake fingerprints

A failure is a known flake ONLY if it matches EVERY field of a fingerprint below.
If any field differs, it is a regression. Stop, do not rerun, and report the
full failure output in the PR. This applies even to tests listed here.

### FP-001
- test: tests/payments/test_webhook.py::test_payment_webhook
- exception: TimeoutError
- message_regex: ^Timed out after \d+(\.\d+)?s waiting for webhook stub on port \d+$
- frame: tests/payments/conftest.py:88 (wait_for_stub)
- conditions: CI only, pytest-xdist workers >= 4
- last_seen: 2026-10-07
- owner: @payments-oncall

### FP-002
- test: tests/checkout/test_checkout.py::checkout_test
- exception: playwright TimeoutError
- message_regex: ^Timeout \d+ms exceeded\..*waiting for frame "payment-iframe"
- frame: tests/checkout/pages/payment.py:41 (wait_for_iframe)
- conditions: CI only, headless chromium
- last_seen: 2026-10-08
- owner: @checkout-team

### Matching rules
- Normalize numbers, ports, and UUIDs before applying message_regex. Nothing else.
- The failing frame must be the exact file:line listed. Same test, different line = regression.
- If you are not in the listed conditions (e.g. running locally, workers < 4), it is NOT a match.
- If last_seen is older than 60 days, treat the fingerprint as expired and ask a human.

The first rule is the one doing the work: no exact match means regression, and the agent stops. That removes the judgment call it kept getting wrong. It doesn't have to decide whether something "seems flaky" anymore. It compares five fields and gets a yes or a no.

Feed the checkout incident through this block and it's over by line two. AttributeError isn't playwright TimeoutError. The agent stops.

Fix 2: no "rerun fixed it" without a receipt

Fingerprints cover the failures you already know about. They don't catch the slow drift, where a fingerprinted flake turns out to fire more often than you thought, or where its cause is a real bug. And they don't survive compaction unless the evidence lives somewhere outside the context window.

So we banned the phrase. "Passed on rerun", "known intermittent" and "flaky, retried" can't appear in a PR summary unless they cite a ledger entry. Every rerun appends one line to a file committed to the repo:

# .flake-ledger.tsv
# date        sha       test_id                                   fp      first_fail_sha256  seed        workers  attempts  outcome
2026-10-06  a41f9c2   tests/payments/test_webhook.py::test_payment_webhook  FP-001  9e1c…b07a         rs=118204   4        2         pass
2026-10-07  d03be71   tests/payments/test_webhook.py::test_payment_webhook  FP-001  9e1c…b07a         rs=55031    6        2         pass
2026-10-08  7c2a9e0   tests/checkout/test_checkout.py::checkout_test        FP-002  41aa…e3c9         rs=902117   4        2         pass

Each field earns its place. The SHA ties the rerun to code a human can check out. The hash of the first failure's output, normalized the same way as the fingerprint regex, tells you whether it's the same failure every time or the test is breaking in new ways. Seed and worker count let someone reproduce the run instead of guessing. Attempts records how many tries the green took.

Here's the CLAUDE.md rule that enforces it:

## Reruns
- You may rerun a test ONLY if its failure matches a fingerprint exactly.
- Before each rerun, append a line to .flake-ledger.tsv. No line, no rerun.
- Max 2 attempts per test per PR. If it is still red, stop and report.
- If a test has 3+ ledger entries in the last 7 days, you may not rerun it at all.
  Stop, and add the label `flake-budget-exceeded` so on-call is paged.
- Never write "passed on rerun" or "known flake" in a PR summary without
  quoting the ledger line(s) you added.

Those thresholds are ours, not universal. Tune them. What matters is the design: claiming a flake now costs something, and after a few claims the agent loses the right to make them. The ledger also outlives the session. When compaction drops the three failures before the pass, the file still has them.

What the first week of the ledger showed

Within days, two of our long-standing "known flakes" blew through the budget. Both had fingerprints. Both had sat on the quarantine list for months. Once all the reruns were on one page, they looked wrong: too frequent, clustered around particular worker counts, the same first-failure hash every single time.

Both were real race conditions. One was in inventory sync, where two workers could reserve the same stock row. The other was in the email digest scheduler, which could double-fire around a DST boundary. The agents had been waving them through with reruns for weeks. We humans had been doing it for a lot longer. The agents didn't invent the habit. They copied ours, ran it more often, and the ledger finally put a number on it.

The agent writes the flakes too

This was the part I didn't enjoy. When we traced our flakes back to their source, a fair few were tests an agent had written. A time.sleep(0.5) waiting on a background job. A fixture that only worked because another test ran first and seeded the database. An HTTP client mock with the right body and the wrong headers, which passed until the retry path kicked in.

So the ledger now feeds back into the rules. When a flake is confirmed and root-caused, fixing the test isn't the end of it. The anti-pattern behind it becomes a ban in CLAUDE.md, with a pointer back to the incident:

## Test-writing bans (each one came from a confirmed flake)
- No wall-clock sleeps to wait for async work. Use the `wait_until(predicate, timeout)`
  helper in tests/util/wait.py. (FP-001 root cause, 2026-10)
- Every test that reads from the DB must create its own rows via a fixture.
  Never rely on data from another test. Run new tests with `pytest -p randomly`
  before opening the PR. (inventory_sync race, 2026-10)
- No naive datetimes in scheduler tests. Freeze time with `freezegun` and include
  at least one case across a DST transition. (email_digest race, 2026-10)
- HTTP mocks must be built from a recorded response fixture, not hand-written
  dicts. (checkout iframe mock, 2026-09)

Every ban is specific and cites where it came from. "Write deterministic tests" is advice, and agents treat advice the way we do: nod, then carry on. "No time.sleep for async waits, use wait_until" is a rule the agent can check its own diff against.

It changed how I think about test hygiene. I used to file it under infrastructure chores, the thing we'd get to after the next release. Once agents are opening PRs against your CI, every flaky test is a training example sitting in their context window. Either you decide what those examples teach, or the flakes decide for you.

The agent wasn't lying in that PR summary. It did what its context window had shown it works.