Three Agents Agreed. They Were Still Wrong.

Last month I ran three agents in parallel on the same ticket: "webhook processor is double-charging customers on retries, add dedup." Same base model, same repo, same prompt template — just three parallel runs. Two of them landed on the same fix. The third disagreed. I went with the majority. The majority was wrong, and I found out from a 3am page from a customer's finance team.

Here's what the 2-of-3 "consensus" produced:

# Agents A and B — the "majority" fix
seen_ids = set()

def handle_webhook(event):
    if event.id in seen_ids:
        return  # already processed
    seen_ids.add(event.id)
    process(event)

And the one the majority voted down:

# Agent C — the "minority" fix
def handle_webhook(event):
    inserted = db.execute(
        "INSERT INTO processed_events (id) VALUES (%s) ON CONFLICT DO NOTHING",
        [event.id],
    )
    if inserted.rowcount == 0:
        return  # already processed, another replica or an earlier attempt got there first
    process(event)

Both pass the same test suite. Both look fine if you skim them in review. The difference only shows up once you're in production: our webhook processor sits behind a load balancer with four replicas. `seen_ids` lives in the memory of whichever pod happens to handle the request. The retry lands on a different pod, finds an empty set, and processes the event a second time. Agent C's version dedupes in the database — the one place all four replicas actually share state. Two out of three agents picked the version that works fine on a laptop and falls over on our real topology. I picked it too, because two agents agreeing felt like confidence.

Majority vote assumes the errors are independent. Ours weren't.

This is the part I got wrong before I was annoyed enough to actually sit down and think it through. Majority vote as a correctness signal — N-version programming, ensembling, the whole "run it a few times and trust the crowd" instinct most of us have internalized — only works because it assumes the errors are uncorrelated. Three independently-built implementations disagreeing, two of them right for unrelated reasons: sure, majority vote surfaces the truth there. That's the entire math behind trusting a majority. Independent attempts fail independently, so agreement means something.

Three runs of the same model on the same prompt are not independent processes. They're one prior, sampled three times. Agents A and B didn't each verify that an in-memory set is safe — they both reached for the simplest idiom that satisfies "add dedup," because that's the highest-probability completion for this class of prompt when nothing in the context window mentions "this runs on four replicas." Agent C didn't get smarter. It just happened to sample a path that surfaced the assumption everyone else was making silently. The 2-of-3 agreement wasn't two independent checks confirming a fix — it was the same blind spot, rendered twice, that I mistook for a second opinion.

That's the trap: consensus launders correlated error into confidence, right up until you notice the correlation. Run the exact same agent five times instead of three, and you'd get 4-of-5, maybe 5-of-5, agreement on the broken version. More "consensus." Same bug. More confidence than you had any right to have.

Better voting doesn't fix this

The obvious next move is to make the vote smarter — weighted scoring, best-of-n with a judge model, self-consistency sampling, an LLM ranking the candidates. I tried a version of it: had a fourth agent read all three diffs and pick a winner. It ranked the in-memory set highest — shorter, no new dependency, reads as "idiomatic." Same blind spot, one layer up. A judge trained on the same distribution as the generators isn't an independent check. It's another vote from a correlated source wearing a different hat.

More samples from a correlated signal don't add information, no matter how you weight them. Best-of-n helps when "n" actually gives you variance in what could be wrong. It buys you nothing when all n share the exact assumption that's broken. You can't out-vote a shared blind spot. You can only make it look more rigorously confirmed.

The actual fix: force disagreement to mean something, then check against the spec, not against each other

Two changes, and they only work together — voting harder on correlated agents is the failure mode, not a step toward the fix.

• ▹Make the agents actually different, not just three processes sharing a name. Give one the deployment topology in its context — replica count, LB config, shared-nothing constraint — and withhold it from another. Run one on a different base model. Constrain one to "no new dependencies," another to "assume horizontal scale." If all three still land on the DB-backed fix despite different constraints, that convergence means something. If they diverge, the divergence is telling you exactly where the ticket was ambiguous.
• ▹Stop asking agents to out-vote each other, and add a reconciler whose only job is diffing every candidate against the spec — the ticket, the architecture doc, the deployment topology — never against the other candidates. In our case the reconciler needed exactly one question: "does this dedup mechanism survive the request landing on a different replica than the retry?" That's a spec question, not a popularity contest, and it kills the in-memory set on the spot without anyone needing to vote.

Look at what the reconciler has that the voting scheme never did: an external source of truth. The vote only ever had the three diffs to check against each other — which is exactly the closed loop that let a shared bias pass itself off as confirmation. The reconciler has the deployment doc. That's the whole trick. Not a smarter aggregator. A different input.

If your agents always agree, you're not getting coverage

Here's the uncomfortable part for anyone running parallel agents on real tickets right now: agreement isn't a signal you get to trust by default, and disagreement isn't noise you average away. If three parallel runs on the same prompt and the same model agree every single time, you haven't bought redundancy — you've paid three times for one opinion and called it validation. Running agents in parallel only pays off if they're wrong in different ways, and they only fail in different ways if you deliberately make them different — different context, different constraints, different models — and then check the output against something that lives outside all three of them. Vote counting was never what made our fix correct. It was what made me feel confident about a fix that a shared blind spot had already decided, one none of the three agents were ever in a position to see.