Posts tagged Distributed Systems — AI × software engineering, made practical.
This lesson maps the eight fallacies of distributed computing onto multi-agent LLM systems, adds a ninth fallacy about the receiver misreading a message, and gives concrete fixes and a pre-ship checklist for both.
You'll learn why wall-clock time differs between machines, where that difference breaks everyday code such as JWT checks, leases, TTLs and last-write-wins, and how leeway, monotonic clocks, Hybrid Logical Clocks and TrueTime handle it, plus how the same problem shows up in AI coding agents.
You'll learn how to record and replay multi-agent runs step by step, covering message ordering, logical clocks, event-sourced traces, exact and counterfactual replay, and how replay results can decide how much autonomy an agent gets.
You will learn what a shard key actually commits you to (locality, load distribution, blast radius and rebalancing cost), how range, hash and directory sharding differ, and how to build in an escape hatch before you need one.
You'll learn why retrying a side-effecting tool call in an LLM agent loop can cause a new action instead of a repeat, and how stable idempotency keys, cancel-and-confirm on timeout, a shared budget and failure classification keep each action to one run.
You will learn how backpressure works in pipelines, why human code review silently degrades when coding agents outpace reviewers, and how to build a review-capacity window that makes agents adapt upstream.
You will learn how service discovery works, how client-side and server-side discovery differ, when a registry is worth it compared with static config, and how the same choice applies to MCP servers and agent tool registries.
You'll learn why payload-based idempotency keys fail when an LLM agent retries, and how to fix it with plan-step IDs, intent keys, convergent tools and guarded one-shot actions.
You'll learn why idempotency keys built from a hash of the payload fail when an LLM agent retries, and how to key side-effecting tool calls on intent (task_id + step_id), with the key minted by the orchestrator.
Learn what CQRS actually splits between write and read models, why projection lag becomes a permanent fact you have to own once you adopt it, and how to expose staleness as data for both dashboards and AI agents.
You'll learn why coding agents working on the same file diverge even with no network partition, and how to design an arbitration rule so merges stop silently picking a winner.
You'll learn how to classify saga and agent-pipeline steps as reversible, semi-reversible, or terminal, and use that classification to decide execution order so your rollback path never tries to undo something that can't be undone.
You'll learn why heartbeat-based liveness checks can't tell a stuck agent from a working one, and how to add a separate progress signal that catches the difference.
You'll learn how gossip protocols spread state across a cluster without a central coordinator, including fanout, convergence math, and gossip-based failure detection.
You'll learn how to spot when a saga across services is actually hiding a bad bounded-context split, and how to test whether a given saga is justified or just compensating for a broken service boundary.
A bloom filter can degrade from 1% wrong to 80% wrong without throwing a single error, and I only found ours because three near-identical tickets landed in the same hour at 2am.
Consistent hashing promises that adding a node only moves 1/N of your keys — nobody tells you what happens in the gap before that node has anything cached, and why the same trap is even nastier in front of an LLM inference fleet.
A rate limiter that never rejected a single request took down four services at once — because 'queue everything' and 'protect the API' turned out to be two very different jobs.
A clean git revert made our dashboard look fine — it was the billing charges, duplicate emails, and spent API credits sitting in other people's databases that told the real story.
Your `max_retries=3` wrapper is quietly promising that a tool call is safe to run more than once, and nothing in your dashboards will tell you when that promise is a lie.
A broker dashboard that says "Exactly-Once: Enabled" isn't lying to you — it's describing something that literally can't exist, and the same blind spot is probably sitting inside your agent's tool-retry logic right now.
The bug wasn't a hallucination and it wasn't sloppy prompting — it was a structural gap between what the agent believed about the saga's state and what was actually true, and no amount of better prompting closes that gap.
If your agent fleet keeps stalling on one PR while everyone else waits, you didn't get unlucky — you sharded your work the same way databases have been getting it wrong for twenty years, and the fix is just as old.
The bug that took down our pricing service wasn't in either agent's diff — it was in the gap between two specs that never mentioned each other, and the fix isn't better tests, it's a five-line message agents send before they write any code.
A sub-agent's "yes, I'm prepared" can quietly evaporate the second its context gets truncated — and most orchestrators have no way to notice until two agents have already clobbered the same resource.
One team's shard key change is a maintenance ticket; the other's is a multi-day outage — and the difference was decided months before anyone noticed the traffic shift.
Consistent hashing was built to make moving a cache key cheap — but an agent session isn't a cache key, and treating it like one turns every routine scale-up into a silent, expensive case of amnesia.
A compensation handler that had never once fired in production finally ran — cleanly, successfully, logged as a win — and quietly refunded the same customer twice, which is exactly the failure mode agentic pipelines are about to hit on a much faster clock.
A user swears their save vanished and nothing was lost — it's a race between your replica and the refresh button, and the same race quietly wrecks AI agents that check their own writes.
The 'flaky' bug where Agent B redoes work Agent A already fixed isn't a timing race at all — it's a consistency choice your context store made for you without asking, and naming it changes everything you reach for next.
Two agents answered the same support ticket with two contradicting replies, no error ever fired, and the postmortem verdict was scarier than a bug: the system worked exactly as designed — nobody had just agreed on what that design was.
Every failure you're currently blaming on prompt engineering — the bottleneck agent, the re-plan that breaks on scale, the handoff where two agents end up disagreeing — already has a name and a fix, and distributed database engineers wrote it up over a decade ago.
A Lamport clock will always tell you which event came first — even when that's a lie — and the fix for it, vector clocks, turns out to be exactly what your multi-agent pipeline needs before a merge quietly erases someone's work.
A vanishing task flag in a two-agent system isn't a merge bug — it's Brewer's CAP theorem showing up uninvited, and your orchestration framework already picked a side without asking you.
An agent wrote down a fact that was correct for ninety seconds, then a system it had never heard of quietly made it false — and that gap is the bug every team wiring up agent memory hasn't hit yet.
A comment vanishes, support says 'data loss,' the logs say otherwise — and the same tiny lag that explains it is quietly wrecking your multi-agent pipelines too.
The tenant_id shard that held up for a year under real human traffic didn't survive three weeks of an unattended AI agent — here's the layer that would have turned the outage into a config change.
Two rows, one identical idempotency key, both attempts legitimate — the 3am postmortem that made us realize our retry wrapper wasn't resending a request, it was re-asking the model what it wanted to do.
A `sleep(random())` hack cut down on duplicate dispatch bugs but nobody could say why — turns out Raft solved this exact standoff a decade ago, and it's missing two pieces the hack never had.
Run more than one coding agent against shared infrastructure and you're not orchestrating a pipeline anymore — you're refereeing a game, and game theory has some uncomfortable things to say about where it settles.
Borrowing Raft's leader election into a multi-agent LLM pipeline makes the design doc look rigorous — right up until you notice it can't stop the one thing that actually goes wrong: two confident agents disagreeing.
A worker agent crashed mid-refund, restarted three times, and charged the same customer three times before anyone noticed — here's why rewording the system prompt was never going to fix it, and what actually does.
Quorum voting only earns its fault-tolerance guarantee when failures are independent — and most "multi-agent" code review setups quietly skip that check before a single vote gets cast.
CAP only kicks in during the rare network partition — PACELC is the quieter, un-diagrammed trade-off charging your agent stack's latency budget on every single request, partition or not.
That double-booking bug you filed under "LLM hallucination" was actually your orchestrator making an unwritten CAP tradeoff — here's how to catch it before the next incident does.
A network hiccup forces every distributed system — including the fleet of LLM calls hiding behind a single "agent" — into a choice between a fast wrong answer and a slow right one, and most teams never notice they made it.
Every replica in your fleet is passing its health check while real requests fail, and once you see why, you'll never trust a green dashboard the same way again.
An on-call team burned two hours blaming a 'flaky' order service, when the real story was two agents independently patching the same order through a gateway that had no way to notice.
One back-of-envelope habit tells you, before you write a line of code, whether your design needs three servers or three hundred — and why your multi-agent pipeline is already slow for reasons that have nothing to do with your code.
Running three parallel agents on the same ticket and trusting the 2-of-3 majority almost cost us a customer relationship — because agreement between correlated agents isn't confirmation, it's the same blind spot showing up twice.