The Retry That Charged a Customer Twice — While Every Log Line Said Success

Day 15: Retries Turn At-Most-Once Into At-Least-Once — Unless You Classify First

Here's the part nobody tells you when you type `max_retries=3` on a tool call: you just made a promise about what happens on the server, not just on the wire. You promised at-least-once delivery for an action that might only be safe to run once. Nobody checks whether the underlying side effect can actually keep that promise, so the bug never crashes. It just charges someone twice and calls it a day.

The double-charge incident

Here's how it actually plays out. Payment tool call goes out. Server processes it, charge succeeds, ledger entry written — done, on their end. The response never makes it back. Network blip, load balancer timeout, doesn't matter which. The agent framework sees no confirmation inside its timeout window and does exactly what you configured it to do: retry. Second call goes out. The server has no idea it's looking at a duplicate, so it charges again. Both calls return 200. Both calls log 'success.' Nothing pages anyone, because nothing failed. The customer found it three weeks later on their statement — not us. That's today's whole point: retries don't fail loudly. They fail as a quiet duplicate that looks, in every log line you'd normally check, exactly like correct behavior.

Quick recap: what Day 14 set up

Day 14 drew the line between at-least-once and at-most-once as delivery guarantees. At-least-once means a call is guaranteed to arrive, possibly more than once. At-most-once means it arrives zero or one times, never duplicated. Neither one is 'better' — it's a trade-off you pick per operation, not a global setting you flip once and forget. Today is what happens when you pick the guarantee for the wrong layer of the system.

The gap: transport retries vs. effect safety are two different layers

Agent frameworks configure retries at the transport layer — LangChain's `Runnable.with_retry()` wrapped around a tool call, OpenAI/Anthropic function-calling loops with a retry decorator bolted on, MCP clients that auto-retry on timeout. That layer only understands network errors, timeouts, malformed responses. It has zero visibility into what the tool actually did server-side. 'Retry this call up to 3 times' is a statement about the wire. It says nothing about the charge, the email, or the row in the database. Those are two separate questions: is it safe to send this request again, and will the response honestly tell you whether the last attempt worked? Most teams answer only the first one, tune backoff and jitter until it feels solid, ship it, and never get around to asking the second.

Why the incident is invisible in your logs

Walk the timeline slow, because this is the part that fools people every time. T0: agent sends the tool call. T1: server processes it, charge succeeds, row written. T2: the response packet drops before it gets back to the agent. T3: the retry timer fires — and from the agent's point of view this looks exactly like a legitimate timeout, because it is one. T4: agent resends the identical call. T5: the server has no memory of T1 being a retry rather than a fresh request, so it processes this one as new too, and it succeeds. T6: the response finally arrives fine. Read back every individual log line — the tool call, the response, the retry count — and it all looks like healthy behavior. There's no error to alert on anywhere. The only trace of the bug is a second row in a ledger that, looked at on its own, passes for a second legitimate transaction. You need a reconciliation job or an angry customer to even find it, because nothing in the request path was ever wrong.

The audit: three buckets, tagged on the tool itself

Before you turn retries on for any tool your agent can call, put it in one of three buckets — and write the answer down somewhere the retry layer can actually read it at runtime. That means the tool schema or its description, not a wiki page nobody opens mid-incident.

• ▹Naturally idempotent: repeating it changes nothing beyond the first call. `get_order_status`, `list_files`, `read_calendar`. Retry freely — there's no extra work to do.
• ▹Idempotency-key-able: the operation has side effects, but a client-generated key lets the server recognize a retry as 'the same request' and hand back the original result instead of repeating the effect. `create_charge`, `place_order`, `provision_resource`. Safe to retry only if that key is generated once per logical intent and carried through every retry attempt.
• ▹Genuinely at-most-once, unsafe to retry: no dedup mechanism exists, and running it again does something new every single time. `send_email`, `post_to_slack`, `create_support_ticket`, `increment_counter`. Retrying here without app-level cooperation just duplicates the effect. These need a human-in-the-loop check, a pre-flight dedup lookup, or retries turned off, full stop.

{
  "name": "charge_card",
  "description": "Charges a customer's card. idempotency: key-able — caller MUST pass idempotency_key; safe to retry with the same key.",
  "parameters": {
    "amount": { "type": "number" },
    "customer_id": { "type": "string" },
    "idempotency_key": { "type": "string", "description": "Stable per logical charge attempt, reused across retries" }
  },
  "x-retry-policy": {
    "safe_to_retry": true,
    "classification": "idempotency-key-able",
    "max_retries": 3
  }
}

That `x-retry-policy` block is the actual fix, not the audit itself. It moves the safety decision out of someone's head and into metadata the retry wrapper checks before it decides to fire again. No tag on a tool should mean one thing: do not retry. Not 'retry 3 times because that's what the framework ships with.'

Why 'just add idempotency keys everywhere' isn't the full fix

Idempotency keys solve bucket two. They do nothing for bucket three, and the instinct to slap a key on every tool and call the problem closed is exactly where teams get burned next. `send_email` has no natural dedup key — the mail server doesn't know or care that two requests are 'the same intent,' it just sends two emails. `post_to_slack` posts twice. `create_support_ticket` opens two tickets that a human now has to notice and merge by hand. Building a dedup key for these needs app-level cooperation you probably don't control: either the receiving system exposes its own idempotency contract, or you build the dedup layer yourself — a store that remembers 'did I already send this exact intent in the last N minutes' before the call ever goes out. That's real infrastructure, not a header you bolt on before lunch. It matters more here than it did for a single backend service, too, because a multi-step agent can retry a whole tool invocation triggered by re-planning after a partial failure — not just resend one HTTP call. The duplicate risk compounds every time the agent loop reconsiders what to do next.

Before you flip retries on

• ▹List every tool your agent can call and classify each one — naturally idempotent, idempotency-key-able, or unsafe-to-retry. No tool ships without this tag.
• ▹For idempotency-key-able tools, generate the key once per logical intent, not per HTTP attempt, and thread it through every retry.
• ▹For unsafe-to-retry tools, default to no retries. If the failure rate is too high to live with, add a pre-flight dedup check or a human confirmation step instead of retrying blind.
• ▹Default new or unclassified tools to no-retry, not the framework's default retry count. An unclassified tool is bucket three until someone proves otherwise.
• ▹Reconcile. For anything that moves money or can't be undone, run a periodic job that checks for duplicate effects — a misclassification will not show up as an error anywhere in your logs.

Day 16 goes deeper on idempotency keys themselves — not as an HTTP header convention copied off a Stripe tutorial, but as a distributed-systems primitive with its own failure modes: key collision, key expiry, and what happens when two agents in a multi-agent system generate keys for what they each think is the same intent, but isn't.

Extend your knowledge

• ▹Stripe's idempotency key documentation — the most widely copied real-world implementation of bucket-two dedup, worth reading even if you're nowhere near payments.
• ▹Go read your agent framework's retry wrapper source — LangChain's `RunnableRetry` behind `with_retry()`, or your MCP client's retry middleware — and check whether it has any concept of per-tool safety, or whether it's just one blanket policy for everything.
• ▹Audit your own production tool list this week using the three buckets above. Most teams find at least one bucket-three tool that's been retrying by accident this whole time.
• ▹Preview Day 16: idempotency keys as a distributed-systems primitive — key collision, expiry windows, and multi-agent key generation conflicts.