Two Years, Zero Failures, One Double Refund: The Saga Rollback Nobody Tested

Why this matters

The forward path of your saga gets exercised every single time an order goes through — thousands of reps a day, bugs sanded down by sheer volume. The compensation path gets exercised exactly once: the day something breaks. In a healthy system, that day might be two years away. Nobody's quietly patching it in the meantime, because it never runs. That silence isn't safety. It's exactly where the rot sits, undisturbed, waiting.

The incident: a two-year-old rollback fires for the first time

Here's the shape of it. An order saga's payment-capture step failed. The compensation handler — written during the original design review, approved, merged, never opened again — kicked in: undo the debit, release the inventory hold. It ran clean. Logged success. Incident closed. Three days later, finance flags a ledger mismatch: the customer had been refunded twice. Nothing threw an exception. The code wasn't broken in any way a stack trace would show you — it was broken because it assumed a ledger state that hadn't been true for two years, ever since the system it was compensating against quietly moved on without it.

Rewind: what a saga actually promises

Quick rewind to Day 10: a saga swaps one distributed transaction for a chain of local transactions, each paired with a compensating action that undoes it if something downstream fails. You trade atomicity for eventual consistency plus a rollback you have to write yourself. The pattern promises that if step N blows up, steps 1 through N-1 get undone. It promises nothing about whether the undo logic is actually correct. Teams conflate 'we wrote a compensating transaction' with 'our system stays consistent under failure.' Those are not the same sentence. The forward path gets proven every second by production traffic. The compensation path gets proven by nothing — until the moment it runs, live, for real.

Why compensation logic rots

• ▹It gets written once — in the design review, to answer 'what happens if this fails?' — and then nobody touches it again.
• ▹Code review checks that it exists and reads plausibly. Almost nobody actually runs the failure path in a real environment before approving it.
• ▹Simulating a mid-saga failure — step 3 of 5, specifically — is a pain to set up. You need to inject the fault at exactly the right point, with the right upstream state already committed. Most test suites skip it: expensive to build, rare enough to fail that it never feels worth the investment. Until it is.
• ▹The forward path keeps evolving — new fields, new invariants, new services bolted on next door — while the compensation path quietly assumes the world still looks the way it did the day it was written.
• ▹There's no forcing function. Nobody gets paged for code that isn't running. Dormant code doesn't show up in a dashboard, an error budget, or an on-call rotation. It shows up in a postmortem.

Anatomy of the incident

Reconstruct the sequence. Debit step: succeeded, ledger entry recorded. Inventory-reserve step: succeeded. Payment-capture step: failed against the provider — a transient decline, not a bug on anyone's part. The orchestrator triggers compensation: reverse the debit, release the hold. The reversal logic fires a brand-new refund transaction. It never checks whether this saga instance had already been refunded, or whether the original debit had already been swept up by a separate, newer settlement job — one added to the system eighteen months after the compensation code was written. Two systems, each correct in isolation, both reaching into the same ledger entry, with no idempotency key tying the compensation back to a specific saga execution. Net result: two refunds for one failed order.

The fix isn't 'write better compensation code'

You can't review your way out of this one, because the bug isn't sitting in the logic you can read on the screen — it's baked into an assumption about state, an assumption that's guaranteed to go stale the moment you stop watching it. The real fix is treating compensation as a first-class path you exercise continuously, not a write-once artifact you file away after the design review.

• ▹Chaos-inject failures on a schedule, in staging: force a step-3 failure deliberately, weekly or on every deploy, so compensation runs against real (staging) data on a loop instead of living as a hypothetical.
• ▹Build idempotency in from the start: every compensating action carries the saga instance ID as its idempotency key, and the handler's first question is 'has this already been compensated?' — not 'did the forward step succeed?'
• ▹Log every compensation firing as incident-worthy, even the clean ones. A compensation that fires successfully in production is rare by definition — treat it as a signal worth a human's eyes, not a line that scrolls past in the log stream.
• ▹Assert against the ledger's actual current state, not the state you assumed when you wrote the code. Compensation logic that reads 'assume the debit exists and nobody else has touched it' is a landmine sitting there waiting for the day another system starts touching that ledger too.

function compensateDebit(sagaId, orderId):
    if ledger.hasCompensation(sagaId):
        log.warn("compensation already applied", sagaId)  # page-worthy, not silent
        return ledger.getCompensationResult(sagaId)

    currentState = ledger.read(orderId)
    assert currentState.status == "DEBITED", f"unexpected ledger state: {currentState.status}"

    result = ledger.refund(orderId, idempotencyKey=sagaId)
    log.event("COMPENSATION_FIRED", sagaId, orderId, result)  # always, success or not
    return result

AI era framing: agents are now writing — and running — your compensation paths

Two things just changed the risk math. First, agentic workflows now orchestrate multi-step, multi-system actions with zero human in the loop between steps — an agent that books a flight, reserves a seat, and charges a card is running a saga, whether or not anyone wrote that word in a design doc. When step 3 fails, it's the agent — or the framework's retry/rollback logic — deciding what to undo, usually with far less scrutiny than a human-written compensation handler ever got. Second, when an LLM is the one writing the compensation branch — and it is, constantly, because 'handle the failure case' is exactly the kind of code people now hand to agents — it tends to produce something that looks locally correct, passes a fast review, and has zero evidence it was ever run against realistic prior state. Same failure mode as the two-year-old handler above. Just compressed from years down to the first incident after a feature ships fast. If you're building agent pipelines that touch payments, inventory, or anything else with side effects, treat every rollback branch an agent writes exactly like inherited saga code: assume it's untested until you've deliberately broken the pipeline mid-sequence and watched the rollback fire with your own eyes. This is where tracing built for multi-agent calls — OpenTelemetry-style spans, LangSmith-style traces — earns its keep: you can actually see the compensation fire in the trace instead of inferring it happened from a clean exit code.

The checklist to take away

• ▹When did your compensation logic last fire in anger — in production, not a test? If the honest answer is 'we don't know' or 'over a year ago,' you're carrying an untested code path in your critical path.
• ▹Is it idempotent against being called twice, and does it check the ledger's current state instead of an assumed one? If a retry or a race calls it a second time, will it notice — or just fire again?
• ▹Who gets paged when it fires? And is a clean compensation run treated as routine noise, or as an event someone actually looks at?

Close

Idempotency and observability fix the technical half of this. The other half shows up the moment the saga crosses a team boundary — your team owns the debit and the ledger, but the compensation depends on inventory state that lives in another team's service, one that changed without you ever hearing about it. Tomorrow: who's actually on the hook for the compensation path when the saga spans two teams that each only trust their own half of it.

Extend your knowledge

• ▹Read the original saga pattern paper (Garcia-Molina & Salem, 1987) for the formal definition of compensating transactions. The idempotency requirement this whole piece leans on is something modern distributed systems bolted on — the original paper doesn't spell it out.
• ▹Go look at your own saga orchestrator's logs — Temporal, AWS Step Functions, or whatever you rolled yourself — and count how many times a compensation activity has actually fired in the last 90 days. If that number is zero, you just found your next action item.
• ▹If you're running agentic pipelines with side effects, check whether your tracing — OpenTelemetry, LangSmith, whatever — captures rollback/compensation spans as their own thing, distinct from forward-path spans. If you can't query 'show me every compensation that fired last month,' you can't audit any of this.
• ▹Set up a recurring staging chaos test that forces a failure at each step of your critical sagas, in rotation. This is a scheduled job, not a one-off manual test you run once and forget.