Your Retry Logic Isn't Buggy — It's Charging Customers Twice

Day 6 — Idempotency Keys: Making Agent Retries Safe

Your retry logic can be flawless. Your orchestrator can be bug-free. Your customer can still get charged twice. That's the part people don't believe until they've watched it happen: duplicate side effects — a second charge, a second email, a second support ticket — don't need a bug to fire. They need a timeout and a retry, and every multi-agent pipeline ships with both by default. I've seen this gap open up in nearly every production pipeline I've built or reviewed, almost always right after the team wires in retries and starts feeling good about resilience. Nothing looks broken when it happens. That's what makes it dangerous.

The concrete failure: a timeout, a retry, two invoices

Walk through it slowly. The orchestrator dispatches a tool call: charge_customer(order_42). The sub-agent's LLM call runs slow — tail latency on inference is just a fact of life — and the orchestrator's timeout fires before any response lands. Except the charge already went through on the payment provider's side. The response simply never made it back across the wire. The orchestrator sees nothing, assumes failure, retries. charge_customer(order_42) fires a second time. Now there are two charges on one order. Nobody wrote a bug. The retry logic did exactly what it was told to do.

Naming the pattern: this is 2015's bug, one layer up

If this smells familiar, it should. It's the exact failure mode that plagued stateless microservices back in the mid-2010s: a caller times out, has no idea whether the callee actually finished, and retries 'just to be safe.' The industry's fix was the idempotency key — a caller-supplied identifier that lets the callee say 'I've already done this one' and hand back the prior result instead of repeating the side effect. Multi-agent orchestration recreated the exact same caller/callee uncertainty — an orchestrator calling into a sub-agent's tools — and somehow the fix didn't make the trip over with it.

Why it's sneakier in agent systems

• ▹LLM call latency swings wildly — a call that normally takes 2 seconds can take 20 under load, so timeouts fire on requests that are actually still succeeding, not failing.
• ▹Retrying on timeout feels like sane self-healing — the same reflex that makes agent frameworks retry on rate limits or flaky errors gets applied blindly, including to calls with real side effects.
• ▹'Stateless sub-agent' quietly gets misread as 'safe to re-run.' Statelessness is about the agent's memory. It says nothing about the blast radius of what it actually does.

Mental model correction: stateless ≠ side-effect-free

Stateless means the sub-agent carries no memory between invocations — every call starts from zero. That's it. It says nothing about whether the tools it calls are safe to run twice. Split every tool your sub-agents can touch into three buckets, and treat only the third one as dangerous:

• ▹Read — get_order_status, fetch_invoice. Run it ten times, nothing changes. Retry freely.
• ▹Write, idempotent by nature — set_status(order, 'shipped'). Repeat it and you land on the same end state either way. Usually fine to retry as-is.
• ▹Side-effecting — charge_card, send_email, create_ticket. Every repeat creates a new real-world effect. This is the only bucket that needs protection.

The fix: deterministic idempotency keys, owned by the orchestrator

Every side-effecting tool call gets a key, derived deterministically from (task_id, step_id) — identifiers the orchestrator already owns and that don't shift across retries. Before firing the call, the orchestrator checks a dedup store for that key. If it's there and the prior attempt succeeded, it just returns the cached result — no re-invoking the tool. One rule matters more than the rest: the key is never generated inside the sub-agent. An LLM has no guarantee of producing the same value twice, so trusting it to mint or preserve the key quietly defeats the entire mechanism.

# Orchestrator-side, not sub-agent-side
key = sha256(f"{task_id}:{step_id}").hexdigest()

# WRONG: hash(task_id + step_id + timestamp)
#   -> every retry mints a new key, dedup never triggers

# WRONG: key = sub_agent.generate_key()
#   -> LLM output isn't guaranteed stable across retries

if dedup_store.get(key) is None:
    dedup_store.mark_in_flight(key)
    result = call_tool("charge_customer", order_id, idempotency_key=key)
    dedup_store.mark_done(key, result)
else:
    result = dedup_store.get(key)  # cached, no re-fire

Pre-deploy checklist

• ▹Every side-effecting tool call carries a caller-supplied idempotency key — the orchestrator generates it, the sub-agent never does.
• ▹The dedup store's TTL outlives your longest possible retry window, including backoff chains and orchestrator crash-and-resume gaps.
• ▹Retries reuse the exact same key derived from (task_id, step_id) — never a fresh key per attempt.
• ▹The dedup store gets queried before the side effect fires, not after — checking post-hoc still lets the duplicate through.
• ▹Failures mid-write are distinguishable from failures pre-write, so a retry after a mid-write crash can resolve to the real outcome instead of blindly re-firing.

Tomorrow: Day 7

Idempotency keys stop the duplicate write. They don't touch the harder question underneath it: who owns the state that decides whether retrying was even the right call in the first place. That's Day 7.

Extend your knowledge

• ▹Read Stripe's idempotency key documentation — it's the reference implementation this pattern is borrowed from, applied to payments API calls.
• ▹Audit your own orchestrator: classify every tool call your sub-agents can make as read / write / side-effecting, and confirm only the last category has key enforcement.
• ▹Check your dedup store's actual TTL against your retry/backoff configuration — most teams set it far shorter than their longest possible retry chain.
• ▹Preview Day 7: state ownership — deciding who has authority to say a retry is even the correct action, not just a safe one.