The Mock That Graded Its Own Exam: How a 95%-Covered PR Double-Charged a Customer

I approved a PR with 95% coverage on a payment-retry module — green check, agent-authored, looked clean. Three weeks later it double-charged a customer in production. I went back to the test file. Every test on the failure path was asserting against a mock — a mock the same agent had invented, in the same PR, to describe how our payment gateway behaves.

What was actually in the diff

Nothing about it read as suspicious — it read as diligent. The agent had written chargeWithRetry, a MockPaymentGateway to stand in for our real processor in tests, and six test cases covering timeout, network error, and the happy path. Coverage on the retry branch: 95%. Here's roughly what the test looked like, reconstructed from the PR.

// paymentGateway.mock.ts — written by the agent, same PR
class MockPaymentGateway {
  charge(orderId, amount) {
    return { status: 'success', transactionId: `txn_${orderId}` };
  }
}

// retry.test.ts — also written by the agent
test('retries charge on timeout and succeeds', async () => {
  const gateway = new MockPaymentGateway();
  jest.spyOn(gateway, 'charge')
    .mockImplementationOnce(() => { throw new TimeoutError(); })
    .mockImplementationOnce(() => ({ status: 'success', transactionId: 'txn_123' }));

  const result = await chargeWithRetry(gateway, 'order_123', 4999);

  expect(result.status).toBe('success');
  expect(gateway.charge).toHaveBeenCalledTimes(2);
});

In review, this clears every visual check a human runs under time pressure: there's a failure case, there's a retry, there's an assertion on call count, there's a green suite. The diff shows implementation and test moving together — exactly what you want from a contributor who's paying attention. Nothing in a unified diff flags that the mock's behavior is fiction. Mocks don't come stamped 'unverified against reality.'

The tell, in hindsight

The mock encodes one belief: a timeout means the charge failed server-side, so retrying is safe. Reasonable guess, reading the signature charge(orderId, amount) — it looks idempotent by intent. It isn't what our actual gateway does. Our gateway can time out on the response while the charge has already landed, and a retry with the same idempotency key comes back with an idempotency conflict carrying the original transaction — not a clean success, not a clean failure either. chargeWithRetry had no branch for that shape. It saw 'not a clean success,' retried with a new key, and charged the customer twice.

The agent never saw that contract. It never talked to the gateway, staging or otherwise — it read the type signature it was calling and the docstring on the retry function, and derived a mock consistent with both. Internally coherent. Just wrong about the world.

Why this isn't an honesty problem

Easy to file this under 'agent cut corners' or 'agent hallucinated a test.' It didn't. The suite genuinely covers 95% of the retry branch, and every assertion genuinely passes. That 95% was never a lie — it's an honest measurement of a suite that faithfully verifies the mock, not the system. Coverage tooling counts lines executed, not behavior verified against reality. It can't tell a test that constrains chargeWithRetry against the real gateway from a test that constrains it against the agent's own idea of the gateway — from coverage's vantage point, those are the same shape of green.

The structural problem is narrower than 'can we trust agents.' When one author writes the implementation, the mock that implementation gets tested against, and the assertions that grade the result — that author is grading its own exam by construction. Doesn't matter if the author is an agent or a rushed engineer at 6pm on a Friday, the failure mode is identical. It just shows up more now, because agents produce that triple — impl, mock, test — fast enough to fit in one PR, at a volume where no reviewer can eyeball every mock body for fidelity to the real dependency anymore.

What we actually shipped at PhoenixDX

The fix isn't 'review agent PRs more carefully' — that doesn't scale, and it already failed me once. The fix is structural: the mock's contract can't come from the same task, or the same agent run, that writes the code being tested against it.

• ▹Contract fixtures get recorded separately — real staging calls against each external dependency, payment gateway, email provider, whatever — and checked in as data. Not generated from a function signature by whatever agent happens to be implementing against it that week.
• ▹An agent building a feature treats the fixture as a given. If the fixture doesn't cover a case it needs — the idempotency-conflict response, say — that's a gap it has to flag out loud, not one it papers over with an invented mock.
• ▹Fixtures refresh on a schedule, pulled from real traffic, so a gateway contract change shows up as its own fixture diff — reviewable on its own terms, independent of whatever feature PR happens to be in flight.
• ▹My review habit changed too. I stopped treating a coverage delta as evidence. Now I open the mock file first. If a mock was authored in the same PR as the code it mocks, by the same agent — that's not a style nit anymore, it's a correctness gate, and I push back on it.

None of this means distrusting the agent's competence. The retry logic itself, once tested against the real conflict response, was a small fix — maybe twenty minutes of work. The expensive part was the three weeks between merge and discovery. That gap existed because the one metric that should've caught it was being produced and graded by the same hand.

The generalizable warning

Coverage is just the instance that burned me. The rule is wider: any metric an agent both produces and grades is a metric to distrust by default. Today it's test coverage on a mocked dependency. Tomorrow it's a benchmark the agent picked for itself, a rubric it wrote for its own PR description, an eval it authored alongside the model change it's evaluating. Same author controlling the yardstick and the thing being measured — the number can be completely honest and still tell you nothing about whether the system works. Before you approve on a metric, ask who owns the ground truth it's checked against. And whether that's the same agent that just wrote the code.