Posts tagged Reliability — AI × software engineering, made practical.
This lesson maps the eight fallacies of distributed computing onto multi-agent LLM systems, adds a ninth fallacy about the receiver misreading a message, and gives concrete fixes and a pre-ship checklist for both.
You'll learn how to structure a multi-agent system as a deterministic pipeline with a single bounded swarm, which can only hand off its output through an exit contract that a code checkpoint enforces.
How to use a durable intent log, idempotency keys built from the run and step, and reconciliation against real systems so you can unwind an agent run that fails halfway through without doubling its side effects.
You will learn how to decide which agent work to drop under overload, using deadline budgets, a shedding ladder, retries at a single layer, retry budgets and priority tiers with a named owner.
You will learn what poison messages are, how dead-letter queues work in SQS, RabbitMQ and Kafka, how to alert on and own a DLQ, how to redrive safely, and what changes when the consumer is an LLM agent.
You'll learn why payload-based idempotency keys fail when an LLM agent retries, and how to fix it with plan-step IDs, intent keys, convergent tools and guarded one-shot actions.
You'll learn how the closed, open and half-open circuit breaker states work, how to pick thresholds and cool-downs, and how to apply a breaker to a coding agent by counting semantic failures and switching the agent to diagnose-only mode.
You'll learn to route each agent step to a cheap or frontier model based on what a mistake would cost and whether anything catches it, rather than on how hard the task looks.
Learn why an LLM-as-judge validator needs its own tests, fail-closed defaults, and separate alerting instead of being treated as trusted plumbing.
You'll learn how to tell whether an agent stuck in a retry loop has a genuine reasoning bug or is just correctly believing a harness that silently swallowed an error.
A rate limiter that never rejected a single request took down four services at once — because 'queue everything' and 'protect the API' turned out to be two very different jobs.
If your ship gate is a single green checkmark, you're not measuring reliability — you're measuring one roll of the dice, and here's the math that proves it.
One 429 turned into a fan-out storm in under ten minutes — here's the assumption baked into every rate limiter that agentic callers blow straight through, and the three fixes that actually hold.