Why an Agent That Reasoned Perfectly Still Refunded a Customer Twice

Day 13 — Saga Pattern: Idempotency, Compensation, and What Breaks When an Agent Runs the Orchestrator

Here's the deal the saga pattern runs on: every step has to survive being run twice, and every rollback has to land on the same result no matter when it fires. For years that deal cost you nothing — a state machine enforced it just by existing. Swap the state machine for something that reasons about state instead of transitioning through it, and the deal comes due.

The incident

A customer got refunded twice. Pull the payment logs and there's exactly one legitimate trigger — the cancellation event fired once, timestamped once, no duplicate delivery anywhere near the queue. The on-call engineer does what you'd do: assumes network retry, at-least-once delivery, a duplicate message slipping past the broker. Dedup keys — clean. Consumer group — clean. The queue did its job perfectly. Something upstream of it decided, all on its own, to ask for the refund a second time.

Rewind: why sagas exist

Day 12 was Two-Phase Commit: a coordinator locks every participant, waits for a unanimous yes, then commits everyone at once. It buys you a genuine ACID transaction that spans services — and the price is holding locks across a network round trip, on every participant, for as long as the slowest one takes. That survives a demo. It doesn't survive a fleet of services with their own deploy schedules and their own uptime targets.

The saga pattern gives up atomicity on purpose, in exchange for never locking anything across a network. Instead of one distributed transaction, you get a chain of local transactions, each committed independently, each with a compensating action defined ahead of time. Step 4 fails? You don't roll back the whole world — you fire the compensations for steps 3, 2, and 1, in reverse order, each one undoing exactly its own local commit. Every step just commits and moves on.

• ▹Orchestration: one coordinator calls every step directly and decides what happens next — including which compensations fire and in what order. Easy to reason about, easy to observe. The catch: that coordinator is now a single load-bearing brain for the whole saga.
• ▹Choreography: no coordinator at all — each service reacts to the previous one's event and emits its own when it's done. More decoupled, sure. But now you're reconstructing the state of the whole saga from event traces after the fact, which is precisely the kind of forensic work you don't want to be doing mid-incident.
• ▹Either style only stays safe under one invariant: every step has to be idempotent — run it twice, get the same effect as running it once — and every compensation has to be deterministic, producing the same undo action no matter the retries, the timing, or who's asking for it.

Why an agent got the orchestrator seat

A pure state-machine orchestrator is deterministic by design — that's the entire point of it. But this team's saga had one step where the downstream response was genuinely ambiguous: a payment gateway that, under load, sometimes comes back slow, partial, or malformed, where the right next move depends on interpreting the response rather than pattern-matching a status code. They wanted something that could look at an ambiguous signal and figure out what it meant, not just dispatch on an exact match. Reasonable thing to want an LLM agent for. It's also exactly the seam where the saga's safety contract gets handed to something that doesn't hold state the way a state machine holds it.

The failure mechanics

This wasn't a hallucination in the sense of inventing a fact out of nowhere. The agent got a real, slow, ambiguous response off the payment step, reasoned about it — plausibly, even — and concluded the refund hadn't actually gone through. So it issued it again. The reasoning wasn't the bug. The bug is that the agent's belief about the saga's state and the saga's actual state were two different things, and nothing forced them back into agreement before the agent acted on it. It broke the idempotency assumption not through malice or randomness, but by confidently holding a wrong belief and acting on it with exactly the diligence you'd want from an engineer who happened to be right.

Why this isn't a prompting fix

The instinct is to patch the prompt: 'always check whether the refund already happened before issuing another one.' That helps at the margin and fails at the tail, for a structural reason. A state machine physically cannot re-enter a state it's already left — the transition table simply has no edge for it. A reasoning agent has no such wall. Its 'memory' of what happened earlier in the saga is inference over context, not a read from a ledger — and inference can always be talked into 'well, given this new ambiguous signal, maybe I was wrong before.' You cannot prompt your way out of that, because the agent isn't malfunctioning when it happens. It's doing exactly what a reasoning system does when handed a genuinely ambiguous input. The fix has to be structural. Language won't hold it.

The actual fix: the log is the state, not the agent

Pull saga state out of the agent's context entirely. The agent doesn't get to be the source of truth about what already happened — an append-only saga log does, keyed by an idempotency key per step. Every transition the agent wants to make gets checked against that log, and recorded in it, before it's allowed to execute. The agent proposes; the log disposes. If the log already holds a committed 'refund issued' record for that key, the second refund call gets rejected at the log layer — no matter how convinced the agent is that it needs to happen.

# saga log enforces idempotency independent of what the agent believes
class SagaLog:
    def __init__(self):
        self._committed = {}  # idempotency_key -> result

    def propose(self, idempotency_key, step_name, action):
        if idempotency_key in self._committed:
            return self._committed[idempotency_key]  # replay, don't re-execute
        result = action()
        self._committed[idempotency_key] = result
        return result

# agent call site
key = f"refund:{order_id}"
result = saga_log.propose(key, "refund", lambda: payment_service.refund(order_id))
# agent's belief about whether refund 'already fired' is irrelevant here —
# the log answers the question, the agent doesn't get to override it

This is the same shape as exactly-once messaging semantics, just moved up a layer. You're not trying to make the agent never wrong about state — you're making it structurally impossible for a wrong belief to produce a duplicate side effect. Compensations stay deterministic because they're keyed and logged the same way: a compensation for a step that was never actually committed is a no-op by construction, not because the agent happened to remember correctly.

The rule to carry forward

If your orchestrator can forget — and any LLM-based orchestrator can, because its memory is reconstructed from context rather than read off a store — your compensations have to stop caring what it believes. Idempotency keys plus an append-only log keep the saga safe independent of the orchestrator's epistemic state, whether that orchestrator is an agent or a machine. Day 14 picks up the same problem from the messaging side: 'exactly-once delivery' is the same guarantee in a different costume, and it fails for the same structural reason — something in the pipeline has to hold the ledger, because nothing that reasons about state can also be trusted to be the record of it.

Extend your knowledge

• ▹Chris Richardson's saga pattern write-up on microservices.io — still the canonical orchestration-vs-choreography breakdown this lesson builds on.
• ▹Temporal's and AWS Step Functions' approach to workflow history: both externalize execution state into an append-only event log, for exactly this reason, long before LLM agents showed up.
• ▹Wiring an LLM into any orchestration role? Audit every place its output triggers a side effect and ask: is there an idempotency key and a durable check sitting upstream of that call, or is the agent's judgment the only gate?
• ▹Day 14 preview: exactly-once delivery semantics — the same 'something has to hold the ledger' problem, viewed from the messaging layer instead of the orchestration layer.