One Customer, Three Refunds: The Agent Bug That Prompt Engineering Can't Fix

Why this matters

Ask a worker agent what step it was on when it crashed. If it can't answer, every retry after that is a coin flip — 'resumed correctly' on one side, 'charged the customer twice' on the other. That coin flip is the real root cause hiding behind most 'flaky agent' tickets you'll see in production.

The incident: 40 minutes of silent restarts

A worker agent was chewing through a batch of refunds: look up the order, call the payment provider's refund API, write a confirmation record, notify the customer. Then it went quiet. On-call watched the log stream sit still for 40 minutes and assumed the model was stuck mid-generation — a long thinking pause, nothing to worry about. It wasn't stuck. It had crashed and restarted three times, and each restart started the task over from the first message in its context, because there was no 'resume' to fall back to — there was no state anywhere to resume from. The agent had no idea it had already called the refund API twice. The payment provider knew, though: three refunds went out for one request.

This is the failure mode that defines this stage of building worker agents: a crash mid-task doesn't pause the worker, it erases it. Read the transcript afterward and it looks perfectly continuous — one long conversation. But to the process itself, every restart is amnesia, not a resume.

The misdiagnosis: this is not a prompt bug

The team's first move was the one everyone reaches for: reword the system prompt. Add a line like 'Before calling the refund API, check if you've already processed this order.' Maybe bold a warning about not repeating actions. It's the default reflex once you've spent months tuning agents through wording — everything starts to look like a language problem, because language is the only lever you've been pulling.

• ▹It fails because the model has no memory across a crash — there's nothing left in its context to 'check against.' You cannot instruct your way out of a missing database row.
• ▹It fails because instructions are probabilistic, not guarantees. Even a well-behaved model will skip the check occasionally — under load, under a strange tool-call ordering, after a context truncation. A payment system can't run on a language model choosing to be careful.
• ▹It fails because the bug was never in what the agent says or reasons — it's in what the surrounding system fails to persist. That's an architecture gap, not a wording gap, and no amount of prompt polish fills in a missing durability layer.

The reframe: a worker agent is a state machine wearing a chat interface

The back-and-forth of messages, tool calls, and tool results is a UI layer, nothing more. Underneath it, a real worker agent moves through the same states any long-running job in a distributed system moves through. The mistake isn't building a state machine — you already have one, whether you meant to or not. The mistake is building it implicitly, in prompt text and conversation history, instead of explicitly, in a store you actually control.

The chat transcript hides this machine, because every restart reads like 'the conversation continuing.' It isn't. When a worker restarts, you're spinning up a fresh process that has to be told — explicitly, from external storage — which state it's in and what's already been done. If that information only ever lived in a conversation history that got wiped by the crash, the state machine has no memory. Which is exactly the incident above.

Checkpoints: persist after steps, not after tokens

A checkpoint is a durable record of 'here's what's confirmed done, and here's what's left.' The granularity that matters is the task's meaningful steps — each tool call with a real-world side effect, or each stage of a multi-stage plan — not every token the model happens to stream. Checkpoint every token and you're burning money on noise. Checkpoint only at the very end and you've built the incident above by design.

{
  "task_id": "refund-8841",
  "state": "checkpointed",
  "steps": {
    "lookup_order": { "status": "done", "result": { "order_id": "ORD-8841" } },
    "call_refund_api": { "status": "done", "idempotency_key": "refund-8841-v1" },
    "write_confirmation": { "status": "pending" },
    "notify_customer": { "status": "pending" }
  },
  "updated_at": "2026-09-10T03:14:02Z"
}

Put that record in a durable store — a database row, not the model's context — and a restarted worker's first move stops being 'start the task' and becomes 'read the checkpoint for task_id, resume at the first pending step.' In the incident, one check — call_refund_api already marked done — would have sent the restart straight to write_confirmation. One refund, not three.

Idempotency keys: make the risky steps safe to repeat

Checkpoints tell you what should have happened. They don't cover the gap between 'the API call fired' and 'the checkpoint recording that fact got written.' Crash inside that gap and a naive retry still double-executes — the checkpoint never had the chance to save. The fix isn't more checkpointing. It's making the side effect itself idempotent, the same way you'd do it in any distributed system: attach a stable key to the operation so the downstream system treats a duplicate as a no-op.

idempotency_key = f"refund-{task_id}-v1"

response = payment_client.refunds.create(
    order_id=order_id,
    amount=amount,
    idempotency_key=idempotency_key,  # same key on every retry
)
# Stripe and similar payment APIs dedupe on this key server-side.
# A retry with the same key returns the original result instead of firing again.

This is the piece most agent frameworks quietly leave out, because tool-calling APIs are built around 'call a function,' not 'call a function exactly once across retries.' You have to generate the key yourself — typically task_id plus a step name plus a version — and thread it through every tool call with a real-world effect: payments, emails, file writes, database inserts. Read-only steps like lookups and searches don't need this; only the ones that change state outside your system do.

Retries and dead-letter queues: give up loudly, not silently

Checkpoints and idempotency keys make retries safe. They don't make infinite retries a good idea. A worker that keeps restarting from the same checkpoint forever, failing the same way each time, is just the original incident replayed in slow motion — instead of triple-charging in 40 minutes, it triple-charges over three days. You need an explicit ceiling.

• ▹Cap retries per task (say, 3 attempts with backoff) and record the attempt count in the same checkpoint record.
• ▹On the final failed attempt, move the task to a dead-letter queue — a durable 'this needs a human' bucket — instead of re-queueing it one more time.
• ▹Page or ticket a human with the checkpoint state attached, so whoever picks it up sees exactly which steps completed and which didn't, instead of re-reading a chat transcript to reverse-engineer it.
• ▹Never let 'escalate to a human' quietly become 'stop retrying and let the task vanish.' A dead-letter entry with no alert attached is just a slower version of the original bug.

Rule of thumb for Day 6

Before you bolt on another tool, another retry wrapper, or another prompt tweak to a worker that's behaving flaky, ask one question: if this crashed right now, what would resuming even mean? If you can't point to a checkpoint record and an idempotency key that answer that, you've found the actual bug — and no amount of prompt engineering was ever going to touch it.

Extend your knowledge

• ▹Read Stripe's idempotency key documentation — it's the clearest public writeup of the pattern and drops straight into any side-effecting tool call your agents make.
• ▹Look at how Temporal models 'activities' and replays event history to recover state, or how AWS Step Functions tracks state machine executions — same checkpoint/resume problem, solved for general distributed workflows years before agents existed.
• ▹Audit one worker agent you already have in production: pick its riskiest tool call — payment, write, send — and check whether it has an idempotency key today. If not, that's your homework for Day 6.
• ▹Read up on the outbox pattern / transactional messaging for the case where your checkpoint write and your side effect can't be made atomic in one step — it's the standard fix for 'the action succeeded but the checkpoint didn't save.'