Your Agent Passed 10/10 Evals. Then It Failed the 11th Run.

Replication: Why "It Passed" Means Nothing for Agents

Here's the uncomfortable version: if your ship gate is one passing run, you're not measuring your agent's reliability, you're measuring your luck on that one sample. Every "green checkmark" eval sitting in your CI pipeline is really one of two things — a probability estimate, or a coin flip that happens to be wearing a lab coat. Most teams have built the second one and are trusting it like the first.

The incident: 10/10, then fail on run 11

Picture the setup: eval suite runs the release candidate against ten scenarios. Ten green checks. Someone hits merge. Two hours later, on-call gets paged — same prompt, same repo state, same everything, and the agent fails in prod. The Slack thread fills up with "but it passed the eval," like passing was some fixed property of the agent instead of the outcome of one particular roll of the dice. Nobody screwed up in the usual sense. The eval suite did exactly what it was built to do. The failure was upstream of that — in what "passed" was ever allowed to mean. It meant n=1.

Name the concept: replication

Replication, in the agent-eval sense, means running the identical (prompt, code, environment) tuple N times and treating the set of outcomes as a distribution — not as N confirmations of one underlying truth. That's a different move from replication in science, where you re-run an experiment expecting the result to hold and you're confirming it does. Here you already know the result won't reliably hold. The point of running it again and again isn't confirmation — it's finding out how often it holds, and what shape the failures take when it doesn't. One agent run isn't "the result." It's a single draw from a distribution you haven't measured yet.

Why this breaks your instincts

Every engineering instinct you own was trained on deterministic software — same input, same output, every single time. "I ran it and it worked" used to be sufficient evidence, full stop, because there was no randomness in the system to hide behind. Agents quietly break that assumption at several layers at once: sampling temperature on the token stream, tool-call ordering when the model has more than one valid next move, retrieval returning a slightly different chunk on a near-tied similarity score, even shell or tool timing changing what state the agent happens to observe mid-task. None of that is a bug you can patch out. It's just how an LLM-driven system runs. Which means a single pass isn't a test result at all — it's a sample size of one from a distribution nobody's characterized.

The compounding math nobody runs

This is the part that actually draws blood in production, because real agent workflows are almost never one step. Say each step in a 5-step chain — read file, plan, edit, run tests, commit — is individually 90% reliable. That's already a strong number, better than most teams' real pass@1. Chain five of those in sequence and your end-to-end reliability isn't 90%. It's 0.9 to the fifth.

0.9^1 = 90%   (single step)
0.9^5 = 59%   (five sequential agent actions)

A "90%-reliable" agent, chained, ships something that
fails almost half the time end-to-end.

A pass@1 number sitting pretty on your dashboard hides exactly this. It tells you whether one step worked once. It says nothing about what happens once five of those steps have to land in sequence — which is the actual shape of most agentic coding workflows on the ground: plan, edit, test, review, merge.

What measuring replication actually looks like

• ▹Run N trials per eval case, not one — same prompt, same repo state, same environment, N independent times
• ▹Report a success rate with a variance (or at least a min/max spread) — not a single pass/fail tick
• ▹Set a real floor: if you can't afford N≥10 on your critical-path evals, what you have isn't an eval — it's an anecdote with a green checkmark stapled to it
• ▹Treat the failing runs as data, not noise. Log the failure mode — the tail is almost always a handful of recurring patterns, not random scatter

The organizational failure mode

Demo culture and eval culture want opposite things from the same agent, and most orgs never notice they're pulling in different directions. A demo rewards exactly one impressive run — you pick your cleanest prompt, your best repo state, and you show the room the run that worked. Eval culture demands the patience to run N trials, sit with a success rate that's a lot less flattering than the demo, and defend a ship decision with a distribution instead of a story. N is slow. N is expensive — compute cost times N, wall-clock time times N, and somebody has to actually build the harness to run and aggregate all of it. So teams default to n=1, and "it worked in the demo" quietly becomes the real acceptance bar, no matter what the eval suite claims to be checking. That gap — between what the eval suite could show and what the team actually bothered to measure — is exactly where the 10/10-then-fails-on-11 incident lives.

Close: from yes/no to how-often

The right question was never "did it work." It's "how often does it work, and what does the tail look like." That reframes shipping an agent from a yes/no call into a statistics call — you're not approving a single behavior, you're accepting a distribution and everything that lives in its tail. Once you're reporting a distribution instead of a checkmark, the next question won't wait: how much variance can you actually live with, and which part of that tail is the part that hurts you. That's tomorrow's concept.

Extend your knowledge

• ▹Read the pass@k metric definition from Chen et al.'s Codex/HumanEval paper — it's the origin of treating code-gen correctness as a sampling problem, not a boolean
• ▹Look at METR's public reporting on agent task reliability — they run repeated trials per task and report success rates rather than single-run scores
• ▹Take one of your own critical-path evals and actually run it N=10 with identical inputs this week — compare the spread to what your current n=1 CI gate reports
• ▹Add a variance or min/max column next to the success-rate column on your eval dashboard, even before you formalize tail-risk gating — visibility first, policy second