Your 3-Agent Review Panel Might Just Be One Judge Mumbling to Itself

Why This Matters

Wire three LLM calls into a majority vote and call it 'more reliable,' and you've built a voting system. Voting systems have known, mathematically predictable failure modes. When the voters aren't independent, the vote doesn't cancel error — it launders it into consensus.

The 2-1 Vote That Went Wrong

Three code-review agents look at the same pull request: same base model, same system prompt, sampled at slightly different temperatures. Two say 'safe to merge.' One flags a race condition in the new concurrency code. Majority rules, the PR ships, and the race condition shows up in production three weeks later.

The majority was wrong, and given the setup, close to inevitable. All three agents are draws from the same distribution — same base model, same prompt template, same diff, same missing context about how this service handles concurrent writes. That's not three perspectives, it's one perspective sampled three times. The dissenting agent wasn't 'less accurate' — it landed on a different tail of the same distribution. The two agents that agreed didn't corroborate each other; they made the identical mistake for the identical reason, because this model, on this class of concurrency bug, tends to under-weight it. Stack three calls to the same judge and take the majority, and the judge's single most probable answer just wins two times out of three. If that answer happens to be wrong, the vote makes you more confident in the wrong answer, not less.

The Hidden Assumption: Wisdom of Crowds Needs Independent Errors

Majority voting and self-consistency (sample k completions, take the mode) both borrow their credibility from 'wisdom of crowds' — the Condorcet Jury Theorem, and Galton's ox-weight-guessing crowd. Aggregate many independent estimates, each a little noisy but unbiased, and the errors cancel, converging on the truth faster than any single estimate could. One word carries the whole theorem: independent.

• ▹Same base model plus same prompt template means same training data, same RLHF shaping, same blind spots — all baked into the weights before a single token gets sampled.
• ▹Temperature randomizes token choice inside the model's existing probability landscape. It doesn't touch what the model doesn't know or systematically gets wrong.
• ▹This isn't bad luck or an unlucky seed. Structurally identical agents produce structurally correlated errors, every time, by construction.

Calling three samples from one model a 'panel' is a category error. It's one judge asked to mumble the same opinion three times, and majority vote is just the loudest mumble.

The Social-Choice Framing: Arrow's Impossibility Theorem

This is social-choice-theory territory, even where the exact math shifts by case. Arrow's impossibility theorem covers ranked preferences over three or more options, and shows no ranked-voting rule can satisfy a small set of reasonable fairness conditions all at once. Our code-review case is actually binary — merge or don't — which puts it under May's theorem instead: simple majority is the provably fair rule there, but only under the same independence assumption the Condorcet Jury Theorem leans on. Either way, the lesson from social choice carries straight over: every aggregation rule encodes assumptions about the shape of its inputs, and those assumptions get violated by design under the wrong input distribution. Translated to agent pipelines: majority vote assumes errors are roughly independent and unbiased, and has no mechanism to detect correlated ones, because detecting correlation was never part of the job. Feed it three correlated votes and it will faithfully, confidently hand you the wrong answer. That's the rule working exactly as designed — on a distribution it was never built for.

This Is the Crack in Day 13's Ensembling

Day 13 covered ensembling and consensus as a way to raise pipeline reliability by combining multiple agent outputs. Naive majority vote is the most exposed aggregation rule in that toolbox, because it weights every vote equally and assumes independence with zero verification. Three calls to the same model with the same prompt isn't an ensemble — it's one model with an expensive, false-confidence tie-breaker bolted on. Ensembling still works. It just needs the diversity majority vote silently assumes you already have.

The Fix: Diversify the Error Source, or Stop Voting

This isn't a tuning knob. It's a structural choice between two fixes, and which one you pick depends on how often the pipeline runs and whether you'll ever get ground truth to calibrate against.

• ▹Diversify the error source: different base models (say, one Claude-family, one GPT-family, one open-weight), different prompt templates or personas (adversarial reviewer, spec-compliance reviewer, security reviewer), even different retrieved context. Worth the extra cost and latency for high-stakes, low-volume calls — a merge-blocking security review, an incident-triage verdict — because you're buying genuinely independent estimates, the one thing that makes voting mathematically sound again.
• ▹Replace voting with a track-record mechanism: keep a running scoreboard of each agent-role's historical precision by finding category, weight votes by that record, or route straight to a specialist or verifier agent instead of counting hands. Worth building when you run the same pipeline shape at high volume and can eventually pull ground truth — CI outcomes, incident postmortems, revert rate — to calibrate the weights.
• ▹What's never worth it: equal-weight majority vote across same-model, same-prompt agents, in either regime. It adds latency and API cost without adding a real reliability signal. You're paying 3x for one opinion dressed up as three.

The Test: Check Whether Your Agents' Mistakes Correlate

Before shipping a voting-based pipeline, don't stop at the ensemble's aggregate accuracy in eval — that number can look great even under heavy correlation, because correlated agents still agree on the easy majority of cases. Correlation hides exactly in the hard, low-margin cases voting was supposed to protect against.

• ▹Pull a sample where you have or can construct ground truth.
• ▹Score each agent individually as right or wrong on that sample — not just the aggregate vote.
• ▹Check pairwise error overlap: are agent A and agent B wrong on the same cases more often than chance predicts? That's correlated error, and it means the vote is decorative.
• ▹Run the removal test: drop one agent — does the aggregate score barely move because it always agreed with another agent anyway? If yes, you're paying for confidence you never earned.

Extend your knowledge

• ▹Read the Stanford Encyclopedia of Philosophy's entry on social choice theory, which covers Arrow's impossibility theorem and the formal fairness criteria it trades off — useful background for framing any aggregation rule you design.
• ▹Read Wang et al.'s 'Self-Consistency Improves Chain of Thought Reasoning in Language Models' (Google, 2022), the paper that popularized majority-vote-over-sampled-completions — read it specifically for what it assumes about independence between samples.
• ▹Run the correlation audit on your current pipeline: for two weeks, log each agent's individual verdict alongside the final vote, then compute pairwise agreement rates between agents on cases with known outcomes.
• ▹Look into mixture-of-agents and 'LLM-as-jury' work that deliberately pairs heterogeneous base models rather than resampling one model, as a concrete example of the diversify-the-source fix in practice.