The Idempotency Key Was Fine. The Retry Was the Bug.

3:47am, on-call, staring at two rows in the orders table. Same order_id. Same idempotency_key. Twenty-nine seconds apart. My first instinct — race condition, or somebody shipped a write path that skips the key check — didn't survive five minutes with the code open. The key was sitting right there. Identical. On both rows. It hadn't failed. Both requests earned it fair and square.

And that's the part the standard playbook has no answer for. "Add idempotency keys" solves a problem this incident wasn't having. The key did exactly what it was built to do. It just wasn't watching the thing that actually changed.

The incident, rewound

Boring setup, on purpose: an agent takes a restock request, reasons about quantities and freight, calls an internal create-order tool. Downstream, a fulfillment service occasionally throws a transient 500 — nothing exotic, connection pool exhaustion under load. The retry wrapper is exactly what you'd sketch on a whiteboard: catch the 5xx, back off, retry up to three times. It did its job. Which is the annoying part — nothing here reads like a bug in the way postmortems usually go.

Attempt one goes out, hangs waiting on the pool, comes back 500. Wrapper catches it, waits, retries. Attempt two fires twenty-nine seconds later and lands — 201 Created. Clean retry, by the book. Except now the orders table has two rows instead of one, and the logs show both attempts as successes.

The problem we thought we'd already closed

We'd been burned by retries before — that's exactly why this stung. Months earlier: the agent occasionally fired the same create-order call twice in quick succession, a genuine byte-identical duplicate, and the fulfillment service happily created two intents for it. Standard fix: an idempotency_key derived from order_id, checked at the request boundary. Shipped, worked, duplicate-intent alerts went to zero, we moved on. That ticket was closed. This was supposed to be the solved problem.

So when on-call opened the DB and saw two rows under one key, the first assumption was regression — a deploy dropped the check, or some new code path routes around it. Nothing had regressed. The key was being checked, correctly, every single time. The digging started only because that easy explanation refused to hold up.

Pulling the logs apart

We pulled both requests from the access logs and diffed them byte for byte. A true retry — the wrapper resending the same bytes — should be identical. It wasn't.

{
  "order_id": "ord_8f21c3",
  "idempotency_key": "ord_8f21c3-create",
- "summary": "Restock: 200 units of SKU-4471, expedited freight",
+ "summary": "Restock order for 200 units of SKU-4471 — expediting after prior submission failed to confirm",
  "total_cents": 184200,
- "fulfillment_intent_id": "fi_9a7e21"
+ "fulfillment_intent_id": "fi_c02b6f"
}

Two fields had moved. The summary got reworded — the agent noticed, in its own context window, that the first attempt had failed, and folded that into a freshly written sentence. Almost charming, honestly. The field that actually mattered was fulfillment_intent_id: a brand new UUID, because nothing in the retry wrapper ever told the agent not to mint one. Here's the mechanism, and it's the whole post in one sentence: the retry wrapper wasn't resending a stored HTTP request. It was re-running the agent's reasoning step and asking it to produce the next tool call. "Retry" meant call the LLM again with updated context — not replay the bytes from attempt one. The model did exactly what it's built to do: synthesize the tool call fresh, from whatever context it currently has. And that context now included "my last attempt failed." A model that reconstructs intent from scratch every time has no obligation to reconstruct identical bytes.

Why the dedupe layer never saw it coming

The request-level idempotency_key was derived from order_id, and order_id never moved between attempts — so at the outer boundary, this looked like a perfectly sanctioned retry. Nobody wrote bad code. But the write that actually mattered — the row in the fulfillment table — was deduped internally on fulfillment_intent_id, the sub-identifier the earlier fix had introduced specifically to stop a different flavor of duplicate. That sub-ID was treated as stable because, in every case anyone had ever tested, it was: a genuine duplicate request carries the same sub-ID, because it's literally the same request. Nobody had modeled the case where the sub-ID itself gets regenerated because the thing generating it isn't a serializer replaying stored state — it's a model reasoning fresh, on every single attempt. The key space the dedupe layer watched (order_id) was rock stable. The key space that actually decided whether a second row got written (fulfillment_intent_id) wasn't. Two different spaces, one silently assumed to track the other, and nothing forcing that assumption to hold.

Where it actually clicked

This isn't a distributed-systems bug. No missing lock, no clock skew, no dual write across services without a saga. Everything downstream of the tool call behaved correctly given its inputs. The nondeterminism lives upstream of the retry, in the step that decides what to send — and no amount of server-side idempotency logic, stacked however carefully, catches a client that regenerates its own intent on every attempt. You cannot dedupe your way out of a client that disagrees with its past self about what it was even trying to do. That reframe is the whole point, because it changes where you go looking for the fix. Server-side dedupe is a distributed-systems tool. This was an agent-architecture problem, and it needed an agent-architecture fix.

The actual fix

Not a smarter hash. Not widening the idempotency key to cover more fields — that's just chasing whichever field drifts next. The fix is refusing to let the agent back into the loop after attempt one. Capture the exact tool call — the real bytes the model produced — the first time it's generated, and freeze it. Every retry replays that frozen artifact, verbatim. The agent's reasoning step runs exactly once per logical operation; the retry wrapper never calls it again.

Mechanically it's a small change: the retry wrapper stops being "catch error, re-run the agent step" and becomes "catch error, resend the stored payload." One thing isn't negotiable — the freeze has to happen before the first network call goes out, not after a success. You're capturing intent at the only moment it's guaranteed to be singular.

The lesson that outlives this one incident

Every agentic pipeline that retries tool calls is making two separate decisions, and this incident only happened because they were sharing one code path.

• ▹"The agent decided to retry" is a control-flow decision — it can live entirely in a wrapper, a queue, a supervisor, with no model involved at all.
• ▹"The agent decided what to retry" is a content decision — and if the model gets invoked again to make it, that decision is non-deterministic by construction, every single time.
• ▹Any architecture where a retry re-triggers the second decision will eventually produce a semantically different write wearing an idempotency key that never changed.
• ▹The fix isn't a bigger dedupe table or a hash over more fields — it's making the second decision structurally impossible to make twice for the same operation.

Idempotency keys assume a retry replays the same bytes. That's a safe assumption for HTTP clients and message queues, because replaying is literally the only thing they know how to do. It stops being safe the instant whatever sits behind "retry" is a model reconstructing its next move from whatever the world looks like right now — including the fact that it just failed. Freeze the bytes before the first attempt ever leaves the building, and the rest of the idempotency machinery works exactly as advertised.