Idempotency keys for agent tool calls

Your agent gets a 504, thinks for a moment and tries again. That seems reasonable, except the first call actually worked, and now a customer has two invoices. Once agents can create invoices, issue refunds or open tickets, a retry isn't only a network event anymore. It's a business event. If you get idempotency wrong, at-least-once delivery becomes charging someone twice, and nothing in your logs will flag it.

Two invoices, one intent

I once had two invoices open side by side. Same customer, same amount, almost the same wording. One said "Consulting services - September". The other said "Professional consulting services for September". The same billing agent created them 41 seconds apart while working on the same task. We had a dedup layer in front of the billing API, and it stayed green the whole time. It didn't crash. It did exactly what we designed it to do, and the design was the problem. This lesson is about why.

Where this fits in the course

Days 19–21 covered timeouts, retries and at-least-once delivery. The key point there was that you can't tell "the request failed" apart from "the response got lost". So you retry, and you accept that the receiver may see the same message more than once. Today covers the other half of that deal. Once you accept at-least-once delivery, idempotency is what keeps it from causing side effects twice.

Here's the definition. An operation is idempotent if doing it twice has the same effect as doing it once. In maths that's f(f(x)) = f(x). In practice, the first call changes the world, and every later call with the same identity does nothing and reports the same outcome. "Set status to PAID" is idempotent by nature. "Create an invoice" isn't. You make it idempotent by giving the operation an identity the server can recognise.

The textbook pattern in 6 lines

key = request.headers.get('Idempotency-Key') or sha256(request.body)
if (hit := store.get(key)):
    return hit.response            # repeat: return what we returned before
store.put_if_absent(key, IN_FLIGHT)  # atomic; a concurrent duplicate waits or gets 409
response = do_side_effect(request)
store.put(key, response); return response

The reference design is Stripe's Idempotency-Key header. The client sends a unique key with a POST. Stripe saves the first result for that key and returns that same result for any repeat. If the key comes back with different parameters, Stripe returns an error so a key can't be reused by accident. Many internal systems skip the client key and hash the request body instead. Both approaches rest on one assumption: a retry sends the same bytes again. That's true for an HTTP library, for a user double-clicking and for a phone flushing its offline queue. Keep that assumption in mind for the next section.

Reconstructing the incident

• ▹t=0s: the agent is on step 3 of a "bill client for September" plan and calls the create_invoice tool.
• ▹The billing service creates the invoice and commits it, but the response is slow.
• ▹t=30s: the API gateway times out and sends the agent a 504. The invoice already exists.
• ▹The agent takes the 504 to mean the call failed. It reconsiders, decides the description could be clearer and writes new arguments.
• ▹t=41s: second create_invoice call. Our dedup layer hashes the payload, gets a new hash, finds no match and lets it through.
• ▹Result: two invoices and no error anywhere. Every component behaved "correctly".

 {
   "customer_id": "cus_XXXX",
-  "description": "Consulting services - September 2026",
+  "description": "Professional consulting services for September 2026",
-  "amount": 4250.00,
+  "amount": 4250,
   "currency": "AUD",
-  "due_date": "2026-10-31"
+  "due_date": "2026-10-31T00:00:00Z"
 }

Look at what changed. The amount is the same number, serialised differently. The date is the same date in a different format. Only the description really differs. A human would say this is obviously the same invoice. To a hash function, the two payloads have nothing in common.

The core insight: agents retry the intent, not the request

When a human or a library retries, it repeats the request. When an agent retries, it repeats the intent. It doesn't resend its earlier message. It writes a new message that it believes does the same thing. Every regeneration can change the wording, the rounding, the number formatting, the field order and whether optional line items are included.

• ▹Network library retry: is the payload identical? Yes, it's the same bytes.
• ▹User double-click: yes, it's the same form state.
• ▹Mobile offline queue flush: yes, it was serialised once and replayed.
• ▹Message broker redelivery: yes, it's the same message.
• ▹LLM agent retry: no. Every attempt is a new generation.
• ▹The agent is the first common caller where the answer is no, and any dedup scheme built on "same bytes" fails against it without telling you.

Fixes that don't work, and why

• ▹Normalise the payload before hashing. You can trim whitespace, sort keys and cast numbers, but you can't normalise meaning. The agent rounds 4249.995 to 4250, splits one line item into two, or moves the due date by a day because it re-read the contract. Each rule you add catches one variation and misses the next one.
• ▹Fuzzy matching, where same customer + similar amount + within N minutes counts as a duplicate. This trades silent duplicates for silent blocking. Customers legitimately get two similar invoices close together, like a deposit and a balance, or two separate projects. Every false positive is revenue you never billed.
• ▹Put "never retry create_invoice" in the prompt. A prompt is a suggestion, not a guarantee. The model may follow it most of the time, but context compaction on a long task can drop it, and a replanning step can decide this case is the exception. You also lose the legitimate retry when the first call really did fail.
• ▹The common thread: all three try to work out identity from the message after the fact. The message is the wrong place to look.

The fix: key on intent, minted by the orchestrator

Your orchestration layer already knows what the agent is doing. It knows which task it's in (task_id) and which plan step it's running (step_id). That pair is the intent, so derive the key from it: idempotency_key = hash(task_id, step_id, tool_name, ordinal). The harness injects that key into every side-effecting tool call, outside the model's control. The model fills in the business arguments and the harness supplies the identity. The tool schema the model sees has no idempotency field, so the model can't forget the key, invent one or change it.

import hashlib

def intent_key(task_id: str, step_id: str, tool: str, ordinal: int = 0) -> str:
    return hashlib.sha256(f'{task_id}:{step_id}:{tool}:{ordinal}'.encode()).hexdigest()

class SideEffectingTool:
    def __init__(self, name, fn, model_schema, store):
        self.name, self.fn, self.store = name, fn, store
        self.model_schema = model_schema   # business args only; no key field

    def call(self, ctx, args: dict, ordinal: int = 0):
        key = intent_key(ctx.task_id, ctx.step_id, self.name, ordinal)

        prior = self.store.get(key)        # durable, same DB as task state
        if prior and prior.status == 'done':
            return prior.result            # duplicate: hand back the original success
        if prior and prior.status == 'in_flight':
            args = prior.args              # replay the FIRST args, not the rewritten ones
        else:
            self.store.insert_if_absent(key, status='in_flight', args=args)

        result = self.fn(**args, idempotency_key=key)   # key also goes downstream
        self.store.mark_done(key, result)
        return result

Two details in that code matter. First, the key also goes downstream, for example as Stripe's Idempotency-Key header. In a case like our incident, the timeout means the wrapper never records "done", so the downstream API is your last line of defence. Second, if an earlier attempt is still in_flight, the wrapper resends the original arguments, not the agent's rewritten ones. That turns the agent's retry back into a byte-identical resend. APIs that reject "same key, different parameters" (Stripe does) then return the first result instead of an error.

Edge cases worth thinking about

• ▹A step that legitimately calls the same tool more than once, like "create one invoice per project". Add an ordinal, such as the loop index or the project ID from the plan. Take it from the plan, not from whatever the model decided to write.
• ▹A replanned step. If the planner throws away step 3 and creates step 3b with a different approach, that's a new intent. A new step_id gives a new key, and that's intentional. Decide on purpose which plan changes keep their step IDs and which get new ones, because that decision is your idempotency policy.
• ▹What to return on a duplicate: the original result, as a success. If you return something like "duplicate rejected", the agent will reason about the error, retry with different arguments or escalate to a human. If you return the original invoice ID, it moves on.
• ▹Argument drift on replay: log it. When the second attempt differs from the first, that shows you how unstable your prompts are, and it's useful when you're debugging.
• ▹Key retention: keep keys at least as long as a task can stay open, including any pauses for human approval. A key that expires before the task finishes doesn't protect anything.

A practical check you can run tomorrow

• ▹List every side-effecting tool your agents can call. That means anything that creates, sends, charges, refunds, books or deletes.
• ▹For each one, ask question 1: who mints the idempotency key? The orchestrator or the harness is fine. The model, the payload hash or nobody is a red flag.
• ▹Then ask question 2: does the key survive the model rewriting the arguments? Test it. Call the tool, simulate a timeout, let the agent retry with reworded arguments, and count the side effects.
• ▹If your answer to question 1 was "the model" or "the payload", you already have this bug. It just hasn't met a slow downstream call yet.

Where this goes next

Day 23 (tomorrow) builds on this. Once every step has a stable identity, you can safely replay a whole workflow after a crash. Steps that already ran return their stored results, and only the unfinished steps actually execute. Durable execution engines depend on exactly this, and so do sagas. If you only remember one line from today, make it this one: identity belongs to the plan node, not the message.

Extend your knowledge

• ▹Read Stripe's API docs on "Idempotent requests", especially the part about a reused key with different parameters, and compare it with what your own wrapper does.
• ▹Brandur Leach's post "Implementing Stripe-like Idempotency Keys in Postgres" goes deep on atomic phases and recovery points. It maps directly onto the in_flight/done states above.
• ▹The IETF HTTPAPI working group draft "The Idempotency-Key HTTP Header Field" shows how the header is being standardised.
• ▹Exercise: pick one side-effecting tool in your agent stack. Add a fake 30-second timeout after the side effect commits and count how many records one task creates. Then add the intent-key wrapper and run it again.