The Double-Charged Invoice Bug With No Bug In It

Day 8 — Idempotency in Agent Systems

Idempotency was supposed to be finished business. Request ID, dedupe table, TTL — ship it, move on. Nobody's argued about this in years. Then agents showed up and quietly reopened the case, because the thing deciding to retry isn't a retry loop anymore. It's an LLM, reasoning its way to "I should probably call this again."

No bug. Just a decision.

A support agent calls `send_invoice`. The call times out. The agent replans, decides the invoice probably didn't go out, and calls `send_invoice` again. The customer gets billed twice. Go looking for the bug afterward and you won't find one — no runaway retry loop, no missing dedupe key, no stack trace to point at. Just an agent that reasoned, correctly given what it knew, that it should try again.

Quick recap: idempotency as a request-layer trick

Pre-agent idempotency lived entirely at the request layer: the client generates an idempotency key, the server checks it against a dedupe table, the key eventually expires. Stripe popularized this pattern, and it's exactly why "just add an idempotency key header" has been boring, solved advice for a decade. It works because the request is a fixed artifact — the same bytes, hashed the same way, whether it's the first attempt or the fifth.

Why agents break that model

An agent doesn't retry a request — it regenerates one. Every pass through the planning loop produces a fresh tool call built from scratch, new arguments assembled from whatever's in the context window right now, with no reference to "the request I sent 30 seconds ago." There's no stable artifact to hash into a key before the tool call exists, because the tool call is synthesized new each time the agent thinks. The client-side idempotency trick assumes a client that remembers building this exact request before. A planning loop doesn't get that memory for free — it gets a transcript, which it may or may not read correctly as "already done."

The confident-narration trap

This is the part that makes it dangerous instead of just annoying. A broken retry loop fails loud — duplicate log lines, a stack trace, a dashboard spike someone already watches. An agent doesn't fail loud. It describes the second call exactly the way it described the first: "I sent the invoice." Nothing in the transcript looks wrong. Nothing to grep for. The output reads as correct because the agent has no concept that it's lying to you — it genuinely believes this is attempt one, because from inside its own reasoning, it is.

What actually has to move

You can't fix this by teaching the agent to "remember better." Prompting it to check its own history is a mitigation, not a guarantee — context windows get summarized, truncated, or just skimmed past. The fix has to move down a layer, off the prompt and into the tool contract itself.

• ▹Tool-side idempotency keys derived from stable domain data (invoice_id, order_id) — not a fresh UUID minted at call time, which defeats the whole point
• ▹A side-effect ledger the tool checks before acting: has this exact operation already succeeded, and if so, replay the recorded result instead of running it again
• ▹Tools that are idempotent by design where you can manage it — "ensure invoice exists" instead of "create invoice" — so a duplicate call is a no-op by construction, not by discipline
• ▹Treat every side-effecting tool definition as a contract question before it ships: what happens on a second call with the same semantic intent

def send_invoice(customer_id: str, invoice_id: str):
    # idempotency_key comes from domain data, not a fresh id per call
    key = f"send_invoice:{invoice_id}"

    if ledger.has_succeeded(key):
        return ledger.get_result(key)  # replay, zero side effects

    result = billing_api.charge_and_send(customer_id, invoice_id)
    ledger.record(key, result)
    return result

Notice what moved: the key isn't generated by the caller — the agent — at request time. It's derived from data the agent already has (invoice_id), and it's checked by the tool itself. The agent can replan, invent a new rationale, call this three times back to back — the side effect still happens once, no matter how many times the model 'decides' to act.

Where this fits in the bigger arc

Strip away the distributed-systems vocabulary and this is a memory problem wearing an idempotency costume. The agent double-calls the tool because it doesn't reliably know what it already did. That's exactly why checkpointing and state management are next on this list: idempotent tools contain the blast radius, but durable state is what stops the agent from having to guess in the first place.

Before you let an agent call anything with a side effect, ask yourself one question: what happens if this runs twice by accident? If you don't have an answer, neither does the agent.

Extend your knowledge

• ▹Read Stripe's idempotency key documentation to see the request-layer version of this pattern done well, then map each piece onto a tool-side ledger
• ▹Audit one side-effecting tool in your current agent stack — payment, email, ticket creation — and write down its actual duplicate-call behavior today. Most teams haven't checked
• ▹Look at how frameworks like LangGraph or Temporal-backed agent runtimes handle durable execution — they're solving the memory/checkpointing half of this same problem
• ▹Tomorrow's concept: state and checkpointing in agent loops — the mechanism that cuts how often an agent has to guess whether it already acted