We Reverted the Bug in 15 Minutes. Cleaning Up What the Agent Already Did Took Days.

Day 15 — Your Rollback Plan Is a Lie the Moment Two Agents Touch the Database

Here's the part nobody tells you when they say "just add rollback": `git revert` only undoes code. The second an agent calls an external API, fires an email, or spends a token, it has changed the world in a way no version control system was ever built to track. Reverting the commit doesn't touch any of it.

The incident: what stayed broken after the revert

At PhoenixDX we had an agent chain that handled signups: create the user record, provision a workspace, call the billing provider to start a trial, send a welcome email, push an event to analytics. One step had a bug — under load it double-provisioned workspaces. We caught it fast and reverted the PR. The diff looked spotless. Fifteen minutes later, support tickets started rolling in. The revert had fixed the code. It had fixed nothing else.

• ▹Duplicate workspaces were still sitting in the third-party provisioning system — that system has never heard of our git history
• ▹The billing provider had already started trial subscriptions for the duplicates — real records, in someone else's database
• ▹Welcome emails had already gone out twice to some users — you cannot un-send an email
• ▹Analytics events were already queued and consumed downstream — dashboards had double-counted signups before we even noticed
• ▹API credits and provisioning tokens for the duplicate calls were already spent — there's no refund path for that

None of that lived in git. All of it lived in systems the revert never touched.

The pattern: code is versioned, the world isn't

This is the idea to actually internalize: version control gives you a perfect, reversible history of your source code, and zero history of the side effects that code caused when it ran. Every agent action that reaches past your own database falls straight into that gap.

• ▹External API calls — third-party systems like payment processors, CRMs, cloud providers accept your writes and offer no "undo" endpoint
• ▹Webhooks fired — once sent, a webhook has already triggered someone else's logic; there's no unfiring it
• ▹Emails and notifications — a delivered message can't be pulled back out of a human's inbox
• ▹Queued and consumed jobs — a job a worker already picked up has already run, no matter what you do to the queue afterward
• ▹Downstream writes by other systems — anything that reacted to your agent and wrote its own state now has to be dealt with on its own terms
• ▹Spent tokens, credits, or quota — money and rate-limit budget gone the instant the call succeeds

Why this bites agentic systems specifically

A traditional deploy has a human sitting between each risky step, and it moves slowly enough that a bad rollout usually gets caught after one or two side effects. Agentic systems break both of those assumptions at once: the agent chains actions on its own, and it does it fast. By the time a human notices something's wrong, the agent hasn't fired one side effect — it's fired N of them, each downstream of the last, each already committed to a system you don't control. A pipeline that provisions, bills, and notifies in three seconds can finish its entire blast radius before your alerting has even evaluated the first threshold. Rollback assumes you catch the failure before the damage compounds. Automating the chain is exactly what removes that assumption.

The fix: undo-safe by construction, not reversible after the fact

The fix isn't a smarter rollback script. It's designing every agent action so that failing mid-chain is survivable *before* the action ever runs. Three things make that true:

• ▹Idempotency keys — every external call carries a unique key tied to the logical operation, not the retry attempt. Run the same step twice, whether the agent retried or a human replayed it, and you get the same result, not a duplicate
• ▹Compensating actions — every action with an external effect ships with its inverse, defined up front: `cancel_trial()` alongside `start_trial()`, `delete_workspace()` alongside `provision_workspace()`. If you can't write the compensating action, the forward action has no business running unattended
• ▹An explicit commit point — the genuinely irreversible steps, charging a card, sending a customer-facing email, sit behind a checkpoint that only fires once every prior step in the chain has confirmed success. The agent plans the whole chain first; the irreversible calls happen last, and only once

# Undo-safe by construction: idempotency key + compensating action
# defined BEFORE the forward action is allowed to run unattended

def provision_workspace(signup_id: str):
    key = f"provision:{signup_id}"          # idempotency key, not a retry counter
    if already_applied(key):
        return get_result(key)               # safe to call twice
    result = provider.create_workspace(idempotency_key=key)
    record_compensating_action(key, lambda: provider.delete_workspace(result.id))
    return result

def on_chain_failure(step_key: str):
    # walk backwards, running each recorded compensating action
    for key in reversed(completed_keys_since(step_key)):
        run_compensating_action(key)

What PhoenixDX changed: the compensating-transaction hook

After the incident, we added a gate to our agent framework: any action tagged `external_effect` can't be registered in an unattended agent chain unless it ships with a paired compensating action and an idempotency key. No compensating action, no unattended execution — the agent stops and asks a human, or the step gets refactored until it qualifies. It's a static check in CI against the agent's action registry, not a runtime nicety we hope someone remembers to add.

The reframe for tomorrow

Stop asking "how do we roll this back?" That question shows up too late — you're only asking it once the agent has already run. The question that actually protects you comes before: "if this fails halfway, what undoes itself, and what doesn't?" That's a design question you answer when you define the action, not a recovery script you write after the incident.

Extend your knowledge

• ▹Read up on the Saga pattern for distributed transactions — it's the classical name for chaining forward actions with paired compensating actions, and it maps directly onto multi-step agent chains
• ▹Look at how Stripe's idempotency-key API works in practice (their docs are the clearest real-world reference for the mechanism) and check whether your own agent's external calls have an equivalent
• ▹Audit one of your production agent chains: list every action tagged (or that should be tagged) `external_effect`, and for each one ask whether a compensating action actually exists today
• ▹If you use a workflow engine (Temporal, Airflow, or a custom queue) under your agents, check whether it already gives you idempotent retries and compensation hooks — you may not need to build this from scratch