Your Agents Aren't Hallucinating — They're Violating CAP Theorem

Day 3 — CAP Theorem Isn't a Label, It's a Policy (and Your Agent Orchestrator Already Made the Choice)

Every design doc I've seen that declares "we're CP" or "we're AP" is describing a decision nobody actually implemented. And every multi-agent system your team shipped this year already made that same decision anyway — by default, in code, without a single line in the design doc.

Quick anchor from Day 1-2

One-line recap: a partition isn't a rare failure mode you insure against — it's the standing premise of any distributed system. The question was never "if." It's "what do you do when." If that doesn't feel obvious yet, go back and read Day 1-2 first — the rest of this won't land.

What 'partition' actually means once you leave the database

CAP was written for replicas talking over a network cable. That's how it gets taught — nodes, latency, dropped packets — so it's easy to walk away thinking it only applies when there's an actual network in the picture. It doesn't. A partition, stripped down to its real definition, is just this: two parts of a system that are supposed to share one truth, holding different views of it, for some window of time. A network is one way to cause that. It is not the only way.

Now look at a multi-agent LLM system. You fan a task out to three sub-agents. Each one reads the shared scratchpad, reasons for a few seconds — a few tool calls — and writes its conclusion back. During that window, every agent is working off a snapshot: not live state, not whatever the other two just decided, just whatever the context window looked like the moment it started. Two agents holding different snapshots of the same shared state, disagreeing about what's true, and nobody finding out until someone reconciles the writes afterward — that is a network partition. There's no cable anywhere in this picture, but the failure shape is identical: divergent views, zero coordination, resolution punted to whoever notices last.

The default nobody chose

Here's the part that should make design reviews uncomfortable. Ask a team "is your orchestration CP or AP" and you'll get a shrug, or you'll get "well obviously CP, correctness matters." Then go read the actual implementation.

• ▹Fan-out-to-sub-agents patterns — CrewAI crews, AutoGen group chats — let every agent write to shared state the moment it finishes. No lock, no version check, no "reject this write if the base snapshot is stale." LangGraph is a partial exception: it throws an error when two branches touch the same state key in the same step with no reducer defined. That stops a silent overwrite, sure — but "crash the graph" isn't a conflict-resolution policy either.
• ▹Shared scratchpad or shared-memory designs — a Redis blob, a vector store, a shared conversation object — get read-then-write from multiple agents with no compare-and-swap, no last-writer-wins rule anyone stated out loud, often no timestamp at all.
• ▹When two writes collide on that scratchpad, most implementations resolve it the same way: whichever write lands last silently overwrites the other. Nobody chose that. It's just what happens when nobody chooses anything.
• ▹That's availability over consistency — every agent keeps working, nothing blocks, nothing errors — chosen by omission, not by architecture. You inherit AP the exact same way you inherit tech debt: by not deciding.

Walkthrough: the double-booking that looked like a hallucination

This is the shape of incident I've now watched happen more than once, so let's walk it like an incident report instead of a diagram. A user asks an orchestrator to book a meeting room and notify the attendee. The orchestrator fans out: Agent A checks calendar availability and books the slot. Agent B, running slightly ahead on a retried step, is still holding its own read of 'slot: open' from a few hundred milliseconds earlier — and books the same slot into a second calendar entry, because its snapshot never saw A's write.

Downstream, the notification agent reads the scratchpad, finds two conflicting 'booking confirmed' records, and — because it's an LLM being asked to produce a coherent answer out of an incoherent context — it doesn't say 'I detected a conflict.' It picks one, narrates it with total confidence, and sends a message referencing a room that's actually double-booked. The postmortem gets written up as 'the agent hallucinated.' It didn't. It was handed two facts that were each true at a different time and asked to reconcile them without ever being told that was its job. That's not a model failure. That's an unacknowledged partition-tolerance failure wearing an LLM costume, because nobody in the room had the vocabulary to name it as anything else.

This is exactly why 'the agent is unreliable, throw in more retries / a bigger model / a stricter prompt' so often fails to fix the actual bug. You're patching the symptom where it became visible — the LLM's confident wrong answer — instead of the cause: two writers, no conflict detection, no ownership rule.

The fix: the same three questions CAP forces on databases

CAP isn't useful as a label ('we're CP') — it's useful as a forcing function, making you answer three questions before the partition happens instead of during the incident review. Apply the same three questions to your agent architecture:

• ▹What counts as a partition here? For agents, define it concretely: two agents reasoning off different snapshots of shared state past some staleness threshold, or two writers targeting the same key/resource without ever seeing each other's in-flight write.
• ▹What do you do once you detect one? Databases pick: block until you can confirm quorum (CP-leaning) or serve the stale read and reconcile later (AP-leaning). Agents need the same choice made explicit — do you halt the second agent's write and force a re-read, or let both proceed and run a reconciliation step before anything reaches the user?
• ▹Who decides which view wins? In Raft it's the leader, elected because it holds the most up-to-date log among the majority. Your orchestrator needs something just as explicit — a designated arbiter agent, a version/timestamp check on the scratchpad, an ownership lock per resource (only the booking agent may touch 'room-state'), or a human-in-the-loop gate for anything with real-world consequences, like a calendar write or a payment call.

None of this requires flipping your whole orchestration into 'CP mode.' It requires that the choice be something your team made and can point to — not a gap the framework's default behavior quietly filled in for you.

Close: your design review question for tomorrow

Next time someone in a design review says 'the orchestrator is AP, agents move fast and we reconcile later,' don't nod along — ask the follow-up: 'okay, so what happens right now when two of your agents disagree about the same piece of state? Walk me through it.' If the honest answer is 'not sure, I think whichever one runs last wins' — congratulations, you just found your next incident before it happened.

Tomorrow, Day 4 picks up exactly here: once you've decided who wins a conflict, you need a mechanism that actually enforces it. That's where consensus protocols come in.

Extend your knowledge

• ▹Read Eric Brewer's own 'CAP Twelve Years Later' — he pushes back on treating CAP as a rigid three-way tradeoff, and the same nuance applies to agent architectures.
• ▹Go read LangGraph's checkpointing/state docs and CrewAI's shared-memory docs specifically for what happens on write conflicts — not the happy path, the conflict path.
• ▹If you run agents against any shared, mutable resource — a scratchpad, a task queue, a calendar — audit it this week against the three questions above, before an incident forces the audit on you.
• ▹Preview for Day 4: read up on Raft's leader-election basics. It's the cleanest concrete answer to 'who decides which view wins,' and it maps straight onto the arbiter-agent pattern from today.