Posts tagged Idempotency — AI × software engineering, made practical.
This lesson maps the eight fallacies of distributed computing onto multi-agent LLM systems, adds a ninth fallacy about the receiver misreading a message, and gives concrete fixes and a pre-ship checklist for both.
You'll learn how the database write-ahead log rule lets a coding agent resume exactly where it crashed, using intent and commit records, fsync, idempotent tools, checkpoints, and safe handling of external side effects.
You'll learn why retrying a side-effecting tool call in an LLM agent loop can cause a new action instead of a repeat, and how stable idempotency keys, cancel-and-confirm on timeout, a shared budget and failure classification keep each action to one run.
How to use a durable intent log, idempotency keys built from the run and step, and reconciliation against real systems so you can unwind an agent run that fails halfway through without doubling its side effects.
You'll learn why payload-based idempotency keys fail when an LLM agent retries, and how to fix it with plan-step IDs, intent keys, convergent tools and guarded one-shot actions.
You'll learn why idempotency keys built from a hash of the payload fail when an LLM agent retries, and how to key side-effecting tool calls on intent (task_id + step_id), with the key minted by the orchestrator.
A clean git revert made our dashboard look fine — it was the billing charges, duplicate emails, and spent API credits sitting in other people's databases that told the real story.
Your `max_retries=3` wrapper is quietly promising that a tool call is safe to run more than once, and nothing in your dashboards will tell you when that promise is a lie.
A broker dashboard that says "Exactly-Once: Enabled" isn't lying to you — it's describing something that literally can't exist, and the same blind spot is probably sitting inside your agent's tool-retry logic right now.
It looks like a mailbox for a partner you trust, but the URL is public — and now agents are wiring these up faster than anyone bothers to check for a signature.
The bug wasn't a hallucination and it wasn't sloppy prompting — it was a structural gap between what the agent believed about the saga's state and what was actually true, and no amount of better prompting closes that gap.
A scheduling agent swore up and down it was March in September — and the root cause wasn't a bad model, it was an idempotency key that never found out the function behind it had changed.
A compensation handler that had never once fired in production finally ran — cleanly, successfully, logged as a win — and quietly refunded the same customer twice, which is exactly the failure mode agentic pipelines are about to hit on a much faster clock.
A payment DAG retried itself clean, the on-call engineer went back to sleep, and three weeks later a double-charge ticket exposed the gap every 'safe' retry hides — the same gap now sitting quietly inside your AI agents' tool calls.
A timeout, a retry, and zero broken code is all it takes for a multi-agent pipeline to double-charge a customer — here's the decade-old fix almost nobody carried over into agent orchestration.
Two rows, one identical idempotency key, both attempts legitimate — the 3am postmortem that made us realize our retry wrapper wasn't resending a request, it was re-asking the model what it wanted to do.
A worker agent crashed mid-refund, restarted three times, and charged the same customer three times before anyone noticed — here's why rewording the system prompt was never going to fix it, and what actually does.
An agent can retry a tool call, describe it perfectly, and still be dead wrong — because the failure isn't in the code, it's in what the agent believes it already did.
An on-call team burned two hours blaming a 'flaky' order service, when the real story was two agents independently patching the same order through a gateway that had no way to notice.