Exactly-Once: Enabled — and the Customer Still Got Charged Twice

Day 14 — "Exactly-Once" Is a Marketing Checkbox, Not a Guarantee

Your queue vendor's dashboard says "Exactly-Once: Enabled." Believe it, skip your own idempotency key, and you're one retry away from charging a customer twice. This lesson shows you exactly which retry does it — and why the broker did nothing wrong when it happened.

The ticket that shouldn't have been possible

2am, on-call ticket. A customer emails support with a screenshot: two identical charges, same amount, ninety seconds apart. The engineer who picks it up pulls up the payment queue's dashboard — the managed broker your team pays for — and right there, in plain text: "Exactly-Once: Enabled." The setting is on. Nobody misconfigured anything. And the customer got charged twice anyway. That gap — between "the feature is enabled" and "the customer wasn't double-charged" — is the whole lesson.

The FAQ page sold you something that can't exist

Pull up almost any managed queue's marketing page and you'll find a version of: "Enable exactly-once processing with a single config flag — no more duplicate messages, ever." That's a paraphrase, not a quote from any one vendor, but the pitch is close to universal. It reads like a toggle. It isn't one. We covered the Two Generals Problem earlier in this series: over an unreliable network, a sender can never tell a lost acknowledgment from a slow one. If the ack for "I processed your message" never shows up, the sender has exactly one rational move — retry — because the alternative, assuming success and not retrying, risks silently losing the message forever. No timeout value, no clever protocol, nothing lets you tell those two cases apart from the outside. That's not a bug in any particular broker. It's a property of asynchronous networks. "Exactly-once delivery," taken literally — the message physically arrives exactly one time — isn't an engineering problem waiting for a smarter fix. It's a claim that contradicts how packet-switched networks work.

What was actually running under the hood

What the vendor actually built — what every "exactly-once" system actually is, once you pop the hood — is at-least-once delivery plus a dedup window on the broker's side. The broker remembers message IDs it's already delivered for some bounded stretch of time, say five minutes, and drops any re-delivery that matches an ID still sitting in that window. That covers the common case fine: a worker times out waiting to send its ack, the broker redelivers a few seconds later, dedup catches the duplicate, nobody notices. It does not cover every case.

Here's what actually happened in the incident. The worker crashed and restarted mid-processing — it had already called the payment gateway and gotten a success back, but died before it could ack the broker. The broker, never having received that ack, queued a redelivery. Except the restart took a little over five minutes — a slow container reschedule — which pushed the redelivery past the edge of the dedup window. The broker checked its memory, found nothing, and correctly concluded: never seen this message ID before. So it delivered it as new. The worker, with zero memory of its own earlier attempt, charged the card again. Every component did exactly what it was built to do. That's what makes this failure mode nasty — there was no bug to file against the broker.

Delivered twice is fine. Applied twice is the bug.

Fixing this starts with pulling apart two things that get flattened into one word — "exactly-once":

• ▹Message delivered twice — the broker hands your handler the same payload more than once. Normal, expected, and free if your handler's built for it.
• ▹Effect applied twice — the side effect itself (a card charged, an email sent, a counter incremented, a downstream API called) runs more than once for what should've been one logical action. This is the actual incident. This is what shows up as a support ticket.
• ▹Exactly-once delivery is unachievable. Exactly-once effect is achievable — but only by making the effect idempotent, never by making delivery more reliable.

The fix: idempotency key at the effect boundary, not the message

The team's actual mistake: they used the broker's own message ID as the dedup key inside the payment handler — same ID, same five-minute lifetime, same blind spot, one layer up. That bought them nothing extra; they'd just rebuilt the broker's dedup window with the identical hole in it. The real fix is a key scoped to the side effect itself — one that lives as long as the charge attempt matters, not five minutes — checked right at the boundary where the effect actually fires: the call to the payment gateway, not the call to the message handler.

# Wrong: dedup on the broker's message ID (same lifetime issue as the broker itself)
def handle(msg):
    if seen_recently(msg.id):   # 5-min window, same blind spot
        return
    charge_card(msg.amount)

# Right: idempotency key scoped to the charge attempt, checked at the gateway
def handle(msg):
    attempt_id = msg.payload["charge_attempt_id"]  # generated by the CALLER, not the broker
    # Unique constraint on charge_attempt_id in the payments table.
    # The gateway/DB rejects a second insert for the same attempt_id —
    # no window, no expiry, enforced by the storage layer itself.
    existing = payments_db.get_by_attempt_id(attempt_id)
    if existing:
        return existing.result
    result = payment_gateway.charge(
        amount=msg.payload["amount"],
        idempotency_key=attempt_id,  # gateway itself dedupes on this too
    )
    payments_db.insert(attempt_id, result)  # unique index enforces the guarantee
    return result

Two things have to both be true here. The idempotency key gets generated once, upstream, at the moment the charge attempt is first created — never regenerated on retry. And it's stored behind a uniqueness constraint at the persistence layer, a real database unique index, not an in-memory cache with a TTL. A cache expires. A unique constraint doesn't.

The same shape shows up in your agent's tool calls

If you're building agentic pipelines, you already have this exact failure mode sitting in your codebase — you just don't have a broker dashboard pointing at it. When an LLM agent calls a tool — "charge the customer," "send the email," "create the ticket" — and that call times out or the agent's process gets interrupted mid-loop, the standard move is to retry the tool call. The model has no memory of whether its first attempt actually landed on the other end; all it knows is it didn't get a confirmed result back. That's the exact same epistemic gap as the broker — a lost response looks identical to a slow one, from where the model is sitting. An agent framework that retries tool calls on timeout (and most do, because otherwise one flaky network blip kills the whole run) is running at-least-once delivery for side effects, with an LLM standing where the queue worker used to stand. If the tool itself isn't idempotent, "the agent retried the API call" and "the customer got charged twice" are the same incident — just with a model in the loop instead of a queue consumer. The fix doesn't change: the tool call needs an idempotency key generated once per logical action, not once per retry attempt, passed through to whatever system actually executes the effect, and enforced with a uniqueness constraint on that system's side — never left to the model's judgment about whether it "already did this."

Tomorrow's question

"Exactly-once" isn't a delivery guarantee you're buying from a vendor. It's a product decision your team makes about where the dedup key lives — scoped to the effect that matters, or to the message that doesn't. Before you trust any "exactly-once" feature — a queue, an agent framework's tool-retry logic, a payment SDK — ask one question: what is the dedup key scoped to, and how long does it live? If the answer is "the message ID, for some fixed window," you don't have exactly-once effects. You have at-least-once delivery with a smaller blast radius — one that stays small enough to slip past testing right up until it's big enough to land you a 2am ticket.

Extend your knowledge

• ▹Go back and re-read the Two Generals Problem lesson from earlier in this series, and map it directly onto your current retry logic — wherever you retry something, ask what you're assuming about a response that never showed up.
• ▹Pick one production queue consumer you own and audit it: is its dedup key scoped to the message ID (broker-shaped) or to the business action (effect-shaped)? If it's the former, this incident is just waiting for its moment.
• ▹Using an agent framework with automatic tool-call retries — LangGraph, a hand-rolled agent loop, whatever? Check whether your tool definitions accept an idempotency key. If they don't, that's the gap to close before you ship any tool with a real side effect.
• ▹Read Stripe's idempotency key documentation. It's the clearest public writeup of scoping a key to an action rather than a request, and it's the direct ancestor of the pattern in this lesson.