Two of Your Agents Just Declared Themselves Boss — Here's the 2014 Paper That Fixes It

Why This Matters

Two agents in your orchestrator both decide, at the same instant, that they're 'the lead.' You don't get a crash. You get something worse: silent double-dispatch, duplicate side effects, and a bug that only shows up under load — which means it only shows up in production, at 2am, and never in your test suite. Raft solved this exact problem for distributed databases over a decade ago. The fix maps almost line-for-line onto your agent coordinator.

The Hook: Two Leaders, One Team

A team I worked with built a multi-agent orchestrator where one agent had to hold 'lead' status to dispatch subtasks to the others. Fine under normal load. Then a slow LLM call, or a retried API request, or a coordinator restart, and — occasionally — two agents both believed they were lead at the same time. Both started dispatching. The same subtask got claimed twice, once with stale state. Nobody could reproduce it reliably, so what shipped was a `sleep(random())` sprinkled in front of the 'become leader' check. It made the bug rarer. Nobody on the team could tell you why it worked, or how much of the problem it had actually solved versus just hidden.

Name the Pattern

That's a split vote. It isn't some novel agent-orchestration problem — it's the leader election problem, and it was solved formally by Raft (Ongaro & Ousterhout, 2014). Day 6 covered why consensus needs exactly one agreed leader before anyone commits state. Today is the mechanism for how that single leader actually gets picked, without the whole cluster deadlocking or flapping between candidates forever.

The Mechanism: Raft Leader Election, Minimum Viable Version

Strip Raft's election down to the four ideas you actually need to fix the orchestrator bug:

• ▹Term — a monotonically increasing counter. Every election bumps it. Any message carrying an old term gets rejected outright — this is what stops a stale leader from being believed after it's already been replaced.
• ▹Election timeout — every follower waits for a heartbeat from the current leader. No heartbeat before the timeout, and it assumes the leader's dead and becomes a candidate.
• ▹Randomized timeout — the detail that actually matters. Each node picks its timeout from a random range (Raft's paper uses 150–300ms), not a fixed value. Candidates almost never time out at the exact same instant.
• ▹Majority vote — a candidate requests votes for its new term and only becomes leader with a majority (N/2+1). A vote is a promise: 'I haven't voted for anyone else this term, and your log is at least as current as mine.'

The `sleep(random())` hack was blindly imitating step three — randomized timeout — without steps one and four. Randomization alone reduces how often collisions happen. It's the term counter plus the majority-vote requirement that makes the outcome actually safe: even when two candidates do collide, only one of them can mathematically win a majority, and every node — including the loser — knows unambiguously who that is.

Why Naive Tie-Breaking Fails

Walk through what the orchestrator actually hit. Every agent used the same fixed short timeout waiting on a leader heartbeat. Under load, the real leader's heartbeat arrived late — not missing, just late. Every follower's identical timeout fired at essentially the same instant. Every one of them concluded 'leader is dead' simultaneously and simultaneously re-requested leadership. No term counter, so there was no way to tell whose claim was newer. No majority requirement, so an agent just declared itself leader the moment it heard its own request — no confirmation from anyone else needed. All of them 'won' at once, dispatched work, collided, and — because the retry logic used that same fixed timeout — did it again next cycle. That's a livelock: no single failure, just an infinite loop of simultaneous, symmetric retries. This is exactly the scenario Raft's randomized timeout exists to make vanishingly rare. Draw each node's wait from a random range and the odds that two expire in the same narrow window collapse fast — and even when it does happen, term-plus-majority resolves it in one round instead of repeating forever.

The Fix, Mapped onto the Agent System

• ▹Term ≈ coordination round number — an integer every agent tracks, incremented on every re-election, attached to every dispatch message so stale claims get rejected on sight.
• ▹Vote ≈ heartbeat ack — a peer agent explicitly acknowledges 'you're lead for round N,' and won't ack a second candidate for that same round.
• ▹Randomized timeout ≈ jittered re-election delay — each agent's 'assume the leader is dead' wait comes from a random range, not a fixed constant, so simultaneous timeouts become the exception instead of the default.
• ▹Majority requirement ≈ don't dispatch until you've got acks from over half the active agents — a candidate that hasn't heard back from a majority doesn't act as leader. Full stop.

Before: fixed timeout, no term, no majority check — an agent self-declares leader the instant its own timer fires. After: jittered timeout collapses the odds of simultaneous candidacy, and even in the rare case two agents do go candidate in the same round, the term counter plus majority-ack requirement means only one of them ever actually dispatches. The other sees it lost the vote and falls back to follower instead of dispatching anyway.

Where the Analogy Breaks

Raft's guarantee rests on one assumption: once a command is committed, every replica applies it and lands on the identical resulting state, because the state machine is deterministic. True for a key-value store. Not true for an LLM agent. Two agents 'applying' the same committed task — same model, same prompt, same committed order — can still produce different outputs, because generation is stochastic. Leader election fixes who gets to decide and dispatch. It does not give you Raft's replication safety, because there's no deterministic state machine underneath reapplying commands identically. You can solve the who's-in-charge problem today and still wake up tomorrow to an inconsistent-results problem. That gap — what 'commit' and 'replication' even mean once the thing executing the command is non-deterministic — is exactly where Day 8 picks up.

Takeaway Checklist

• ▹Does your coordinator use randomized backoff/timeout for re-election, or does every agent wait the same fixed duration?
• ▹Does it have an explicit term/epoch/round number attached to leadership claims, so stale claims get rejected instead of silently believed?
• ▹Does a candidate require majority acknowledgment before it starts acting as leader, or does it act the instant it decides to?

Extend your knowledge

• ▹Read the original Raft paper, 'In Search of an Understandable Consensus Algorithm' (Ongaro & Ousterhout, 2014) — the leader election section is short and directly readable.
• ▹Play with the Raft visualization at raft.github.io to watch randomized timeouts and split votes happen live.
• ▹Check whether your orchestration framework (e.g. anything built on top of a task queue or actor model) exposes a term/epoch concept at all — many don't, and it's worth knowing before you hit this bug in production.
• ▹Come back for Day 8, which picks up where this lesson stops: what 'consistent replication' even means once the thing executing the committed command is a non-deterministic LLM agent.