The 2 A.M. Bug Where Nothing Crashed — and the System Still Did Exactly the Wrong Thing

Day 8 — Consistency Models: The Guarantee You Never Wrote Down

You picked 'eventual consistency' the way almost everyone does: it was the default in the NoSQL client, the docs made it sound harmless, and nobody ever made you say out loud what that word actually guarantees. That's fine right up until two of your agents touch the same row in the same second. Then the gap between what you assumed and what you shipped stops being theoretical and starts being an incident.

The vanished write

2 a.m., on-call phone buzzing. The setup: a router agent claims a support ticket by writing a 'claimed_by' record, then a worker agent picks it up and calls the LLM to draft a reply. That night, two workers picked up the same ticket. Two different drafts, both sent to the customer four minutes apart, flatly contradicting each other.

Nothing crashed. No exception fired, no write failed — the claim record landed just fine. The second worker simply read the ticket, saw it as unclaimed, and claimed it too. The postmortem verdict was almost worse than finding a bug: the system had worked exactly as designed. Nobody had ever agreed on what that design actually was.

What everyone assumed vs. what the code actually promised

What the team believed: once an agent claims a ticket, every other agent sees that claim instantly. That's a strong-consistency assumption — read-after-write, everywhere, no exceptions, no asterisks.

What the code actually did: write the claim to the primary, then let worker agents read from a regional replica — a choice made months earlier to keep p99 latency low on the hot path that gates every LLM call. Replication lag usually sat under 100ms, which is invisible in testing, invisible in staging, and invisible across ten thousand ordinary ticket claims. It only turns visible the instant two agents poll inside that 100ms window — and with a fleet of idle workers all waking up the moment an LLM call returns, that happens more often than a p99 number lets you believe.

• ▹Assumed guarantee: strong consistency on claim state, enforced everywhere.
• ▹Actual guarantee: eventual consistency on the read path, strong consistency only on the write path.
• ▹Neither one had ever been written down. Both lived as unspoken beliefs until the incident dragged them into the light.

The guarantee ladder, in plain terms

Here's the actual payload of today's lesson: four rungs, one sentence each. You should be able to place any system you own on this ladder without opening a doc.

• ▹Strong consistency: every read sees the latest write, everywhere, immediately. You pay for it in latency and coordination — a read might wait on a quorum or a lock. Reach for it when stale means wrong: money moving, an agent deciding whether a refund already went out.
• ▹Causal consistency: reads respect cause and effect, even when unrelated writes land in any order. Agent A writes 'ticket escalated,' agent B writes 'reply sent because escalated' — every reader sees the escalation first. Two unrelated tickets updating at the same moment make each other no promises at all.
• ▹Read-your-writes: a client always sees its own writes on the next read, and that's the entire promise — nothing is said about what other clients see. This is the one that bit us: the router agent that wrote the claim would've seen it instantly. The worker agent reading it was a different client, and the guarantee never covered it.
• ▹Eventual consistency: with no new writes, every replica converges — eventually. No promise on when, none on ordering until then. It's the cheapest, fastest rung, and the right call for a lot of agent state, provided everyone actually agreed that's what they were buying.

Why the gap stays invisible until it doesn't

Nobody wrote 'this system is eventually consistent' in a design doc, because nobody ever decided it — it was inherited. The read-replica setting sat in a client SDK, tuned months earlier for LLM-call latency by someone who'd never heard of the claim-record feature that would later get built on top of it. In a multi-agent system this bites harder than in a normal web app, because agents don't pause to double-check the way a human would. They act immediately, confidently, on whatever state they just read — in parallel, at machine speed. A stale read isn't a rendering glitch a user shrugs off and refreshes past. It's an autonomous decision, already executed, already irreversible, milliseconds after the state went stale.

The fix that actually shipped

We didn't flip claim reads to strong consistency across the board — that adds a synchronous round-trip to every worker's poll loop, at fleet scale, right on the path that gates every LLM call. What we shipped instead was a one-page contract, three parts:

• ▹What we promise: claim writes are read-your-writes for the writer immediately, and eventually consistent for every other reader within a documented 250ms bound.
• ▹What we don't promise: no ordering guarantee across unrelated tickets, no strong consistency on the read replica, and workers may not treat 'unclaimed' as authoritative inside that 250ms window.
• ▹Who signs off when it changes: the on-call lead and the agent-platform owner both have to approve any change to the replication lag bound or the read path — it's a shared, load-bearing promise now, not a config flag someone can quietly bump.

The code diff was almost nothing: workers now do a second claim-check after a short delay, respecting that 250ms bound, before they burn an LLM call on expensive work. The real fix wasn't the delay. It was that the number finally had a name and an owner.

The rule to take away

Before you pick a consistency model, write the guarantee down as one sentence a support engineer could read back to an angry customer — something like: 'once an agent claims a ticket, other agents may still see it as unclaimed for up to 250 milliseconds.' If you can't write that sentence, you haven't chosen a consistency model. Something upstream already did — a client library default, a DB flag, a caching layer — quietly, on your behalf, and it will introduce itself to you at 2 a.m. as somebody else's incident.

Where this fits in the arc

Tomorrow builds straight on top of this one — it assumes you can already say your system's guarantee in that one sentence, cold. Today's the last day you get to not know it.

Extend your knowledge

• ▹Read the original Amazon Dynamo paper (2007) for where 'eventual consistency' entered mainstream systems design — it also cites Terry et al.'s earlier work on session guarantees, which is where 'read-your-writes' actually comes from.
• ▹Kleppmann, Designing Data-Intensive Applications, Chapter 5 ('Replication') — the clearest plain-language treatment of the consistency ladder.
• ▹Check your DB or cache client's actual default (e.g. DynamoDB's ConsistentRead flag, Redis replica-read settings) — most teams have never looked at what it's set to.
• ▹Run a Jepsen-style test (or read a published Jepsen report) against your own datastore to see the actual staleness window under load, instead of assuming the docs number holds.