Your 5-Agent Code Review Quorum Might Be One Agent Wearing Four Costumes

Day 5: Quorum-of-agents isn't quorum consensus

Five agents reviewed your diff. Five agents approved it. Before you write "unanimous, 5/5" in the PR description, ask yourself something uncomfortable: were there ever five reviewers, or was it one reviewer wearing four costumes? Teams are paying 3x–5x inference cost to run "quorum" code review across multiple LLM calls, betting that voting buys them Raft-style fault tolerance. Most of the time it doesn't — the one precondition that makes voting actually work is missing, and nobody checked for it.

The same-cutoff test

Try this on your own pipeline before you trust it. Take one diff, plant one subtle bug in it — a timezone-naive datetime comparison, an off-by-one in a pagination cursor, whatever your team actually gets wrong. Fire it at 5 independent calls of the same base model, same prompt, temperature bumped above 0 so they're not deterministic clones of each other. Now count how many catch it.

• ▹In my runs, the five instances never scatter across different mistakes. They either all catch the bug, or all miss it — together, like they'd agreed on it beforehand.
• ▹The reason isn't mysterious: they share a training cutoff, a pretraining corpus, and the same base model's gut instinct for what "looks like a bug" in a diff.
• ▹That's the exact failure mode as a Raft cluster where all 5 nodes plug into one power supply. You counted 5 replicas. You bought 1 unit of actual redundancy.

Recap: independent failure, from Day 1–4

Quorum systems like Raft and Paxos only tolerate f failures out of 2f+1 nodes on one condition: the failures have to be statistically independent. One node crashing can't tell you anything about whether the node next to it is about to crash too. That's the whole reason you rack replicas on separate power circuits and separate availability zones — correlated infrastructure kills the math before a single vote gets cast.

The bolt-on: quorum-of-agents as a product pitch

A wave of multi-agent code review and coding products now pitch "3 agents vote, majority wins" as a reliability feature, lifting the vocabulary of consensus systems almost word for word — quorum, majority, fault-tolerant. It sounds reasonable: more eyes, fewer misses. What's missing is any check that those 3 or 5 "eyes" actually fail independently. In practice they're usually 3 calls to the same model family, same system prompt, sometimes the same context window, differing only in sampling noise. Nobody asks the independence question, because the Raft analogy makes the voting step feel like the guarantee comes bundled in. It doesn't. The guarantee was earned by the infrastructure design — not by the act of counting.

The math: correlated errors don't shrink with more voters

Model each reviewer as a coin with error rate p on a given bug class. Independent voters: majority-of-3 is wrong only if at least 2 of 3 miss the bug. Correlated voters: if the miss comes from a shared blind spot, all 3 miss it together, every single time — the vote never gets a chance to override the shared bias.

p = 0.20 (single-agent error rate on this bug class)

Independent agents, majority of 3 wrong requires >=2 wrong:
P(majority wrong) = C(3,2)*p^2*(1-p) + p^3
                   = 3*(0.04)*(0.8) + 0.008
                   = 0.096 + 0.008
                   = 0.104
=> error drops from 0.20 to ~0.10. Quorum earned its cost.

Fully correlated agents (same base model, same blind spot):
All 3 vote the same way on this bug class, every time.
P(majority wrong) = p = 0.20
=> majority-of-3 correlated ≈ majority-of-1. You paid for
   3x inference and bought 0 reduction in error rate.

Real deployments land somewhere between these two poles. But the direction is what should worry you: the more the voters share training data, base model, and prompt framing, the closer the quorum's error rate sits to the single-agent error rate — no matter how many instances you spin up.

Where quorum still earns its cost

None of this makes multi-agent review worthless. It means the diversity has to be real, not decorative. Real diversity decorrelates the failure mode. Cosmetic diversity just relabels the same model three times and charges you for it.

• ▹Real diversity: different base model families — one Claude reviewer, one GPT, one Gemini. Different pretraining data means different blind spots.
• ▹Real diversity: different context or tools per agent. One only sees the diff, one also runs the test suite, one gets repo-wide grep access. Different inputs, different errors.
• ▹Real diversity: adversarial roles. Tell one agent to hunt security issues, one performance, one correctness. Splitting the search space decorrelates by construction.
• ▹Cosmetic diversity: same model, same prompt, different temperature or seed. Mostly resamples the same distribution — you get a narrower confidence interval on the same bias, not a different bias.
• ▹Cosmetic diversity: "self-consistency" dressed up as a second opinion. It's the same reviewer talking to itself three times and calling it a committee.

A cheap diagnostic: the seeded-bug test

Before you trust a quorum setup in production, run this once. It costs one bad diff and five review calls.

• ▹Pick 5–10 bug classes your team actually cares about — off-by-one, race condition, missing null check, wrong timezone handling, SQL injection via string concat.
• ▹Seed one instance of each, one at a time, into a clean diff, and run your quorum against each seeded diff several times.
• ▹Compute the observed agreement rate: how often do the voters split, versus how often do they move in lockstep — all catch it, or all miss it?
• ▹Compare that against what independence predicts for your measured per-agent error rate, using the binomial formula above. If observed lockstep runs far higher than the independent-model prediction, your quorum is correlated, and the vote isn't buying you the safety margin you think it is.
• ▹Do this per bug class, not once overall. Correlation is usually bug-class-specific — every model in your quorum might miss the same class of concurrency bug while scattering normally on naming and style issues.

The real cost is false confidence

The inference bill for a 3x or 5x quorum is real, but it's small change next to the actual cost: a merged PR that reads "reviewed by 5 agents, unanimous approval" carries more social and organizational trust than a single review — earned or not. Correlated voting doesn't just fail to catch the bug. It manufactures consensus-shaped evidence that the bug isn't there. That's worse than no review at all, because it suppresses the human double-check a single, less-confident AI comment would have triggered. Tomorrow, Day 6, we go past vote-counting into what actual fault tolerance requires when your "nodes" are agents: detecting and modeling failure correlation directly, instead of assuming it away.

Extend your knowledge

• ▹Re-read the failure-independence assumptions in the Raft paper — Ongaro & Ousterhout, 'In Search of an Understandable Consensus Algorithm.' The same paper spells out why replica placement, not the voting algorithm, is what buys you fault tolerance.
• ▹Run the seeded-bug diagnostic from this lesson against whatever review or coding agent setup you run in production right now, before you trust its 'multiple agent' claims.
• ▹Look into mixture-of-agents and LLM-ensembling research that measures inter-model error correlation directly, instead of assuming diversity from sampling alone.
• ▹Tomorrow, Day 6: what real fault tolerance requires for agent fleets once you move past vote-counting — modeling and monitoring correlation directly.