Sagas for agent runs

The agent got through 4 of 7 steps, then died. I ran `git revert`, and it cleaned up the one step that didn't matter. The other three had already left the repo: a migration on shared staging, a flag live for a QA cohort, and a Jira transition that fired a webhook into the client portal. Two people had already acted on those signals. Cleanup took three hours. Git couldn't help with any of it.

On paper it was a routine task. Add a column for a new billing field, put the UI behind a flag, turn the flag on for the QA cohort, move the ticket to In Review, let CI run, then post a summary in the team channel and ping the partner's sandbox API so their side could start testing. Seven steps. I'd watched the agent do runs like this a dozen times. This time CI hit a flaky integration test, the runner timed out, and the agent's session ended mid-retry.

I opened the branch, saw a half-done change, and reverted it. Clean branch. For about ninety seconds I thought we were fine. Then QA asked in Slack why the new billing panel was showing up for their test accounts, and whether that meant it was ready. A few minutes later our account lead forwarded a screenshot: the client portal showed the ticket as 'In Review', and the client had already asked when they could take a look.

The revert had fixed the part of the run nobody was looking at.

The blast-radius audit

Afterwards I sat down and wrote out where each step's side effect actually ended up, and who reads it. If you run any agent workflow more than once, do this.

#  Step                      Where the effect lives          Who reads it            Undo cost
1  create branch             repo                            machine (CI)            free (delete ref)
2  run migration             shared staging DB               machine + other devs    down-migration, if it exists
3  enable flag for QA        flag service                    humans (QA cohort)      restore old value, if you saved it
4  transition ticket         Jira + webhook -> client portal humans (client, AM)     reverse transition, but the webhook already fired
5  run CI                    CI runner                       machine                 none needed
6  post summary              Slack                           humans (team)           delete is possible, being read can't be undone
7  notify partner sandbox    external API                    machine + their humans  depends entirely on their API

Only one row has a free undo: the repo. Most of what the agent actually does happens somewhere else. Yet most of the tooling we've built around agents treats the repo as the system of record: diffs, PR review, revert. The real consequences show up in shared databases, flag services, ticket trackers and chat. That's where humans pick up signals and act on them.

A multi-step agent run that touches four systems is a distributed transaction. We just weren't treating it like one.

Why the usual fixes don't work

**git revert covers one row of the table.** Nothing more to say.

**Retries are worse with agents than with normal code.** Retry an HTTP call and you send the same request again. Retry an agent step and you don't: the LLM sees the failure in its context and improvises a new attempt. I'd seen exactly this in logs from earlier runs. The branch already exists, so it creates `feature/billing-field-2`. The migration 'failed', so it writes a second migration file with a new version number that adds the same column under a guard. It decides the earlier Slack message was premature and posts a 'correction'. Every retry piles on more side effects instead of converging on one state.

**Asking the agent to undo its own work doesn't work either.** I tried it during the cleanup: fresh session, the transcript, and the instruction 'reverse everything you did'. It confidently listed steps, some of which never ran, and missed the flag change entirely, because that tool call had been summarised away in a context compaction. What the agent remembers doing is a story it tells, not a record. You can't run compensation off a story.

So the fix belongs in the harness, not the prompt. None of it is new. It's the saga pattern from Garcia-Molina and Salem's 1987 paper, plus the vocabulary Chris Richardson uses on microservices.io (compensatable, pivot and retriable transactions), applied to tool calls.

Step 1: Pair every tool with a compensation

Every capability in the tool registry is a triple: `do`, `undo`, `verify_undone`. Write the undo when you write the tool, not on the day you need it. `verify_undone` exists because undo calls fail too. Check the real system. Don't trust a 200.

@dataclass
class Capability:
    name: str
    kind: str                      # 'reversible' | 'compensatable' | 'irreversible'
    do: Callable[..., Effect]
    undo: Callable[[Effect], None] | None
    verify_undone: Callable[[Effect], bool] | None
    probe: Callable[[dict], Effect | None] | None = None   # find the effect from args alone


def create_branch(repo, name, base):
    sha = git.create_ref(repo, f'refs/heads/{name}', base)
    return Effect(repo=repo, ref=f'refs/heads/{name}', sha=sha)

REGISTRY['create_branch'] = Capability(
    'create_branch', 'reversible', create_branch,
    undo=lambda e: git.delete_ref(e.repo, e.ref),
    verify_undone=lambda e: not git.ref_exists(e.repo, e.ref),
)


def run_migration(db, version):
    before = schema_hash(db)          # taken BEFORE the migration
    migrator.up(db, version)          # must ship with a tested down()
    return Effect(db=db, version=version, schema_before=before)

REGISTRY['run_migration'] = Capability(
    'run_migration', 'compensatable', run_migration,
    undo=lambda e: migrator.down(e.db, e.version),
    verify_undone=lambda e: schema_hash(e.db) == e.schema_before,
)


def set_flag(flag, segment, value):
    snap = flags.snapshot(flag)       # taken BEFORE the write
    flags.set(flag, segment, value)
    return Effect(flag=flag, snapshot_id=snap.id)

REGISTRY['set_flag'] = Capability(
    'set_flag', 'compensatable', set_flag,
    undo=lambda e: flags.restore(e.flag, e.snapshot_id),
    verify_undone=lambda e: flags.matches_snapshot(e.flag, e.snapshot_id),
)


def transition_ticket(key, to_status):
    prev = jira.status(key)
    tid = jira.transition(key, to_status)
    return Effect(key=key, prev=prev, transition_id=tid)

REGISTRY['transition_ticket'] = Capability(
    'transition_ticket', 'irreversible',   # the webhook makes it so, see Step 3
    transition_ticket,
    undo=lambda e: (jira.transition(e.key, e.prev),
                    jira.comment(e.key, correction_note(e))),
    verify_undone=lambda e: jira.status(e.key) == e.prev,
)

A few things I learned writing these for real:

• ▹`run_migration` only counts as compensatable if the down-migration exists and has actually been run at least once. Your migration tool may let people skip `down()`. The agent's version of the tool shouldn't.
• ▹A `set_flag` without a snapshot can't really be undone. 'Turn it off' and 'put it back' aren't the same thing, because the flag may already have been on for another segment before the agent touched it.
• ▹`transition_ticket` can undo the Jira state, but nothing unfires a webhook. That's why it's tagged irreversible, even though Jira will happily let you move the ticket back.

Step 2: Log every effect before the call runs

The harness writes an append-only saga log that lives outside the agent's context. Before each call it records an intent with every identifier it can allocate up front: branch name, migration version, flag snapshot ID, idempotency key. After the call it records what came back: the ticket transition ID, the Slack message `ts`, the external API's request ID.

People skip the write before the call, and that's the one that saves you when the process dies mid-call. An intent record with no completion record tells you the call might have happened, and `probe` plus `verify_undone` let you go and check.

def invoke(run_id, cap_name, **args):
    cap = REGISTRY[cap_name]
    key = idempotency_key(run_id, cap_name, args)
    if (prior := log.find_done(run_id, key)) is not None:
        return prior.effect                           # same intent: converge, don't redo
    log.append(run_id, 'intent', cap_name, args=args, key=key)
    effect = cap.do(**args, idempotency_key=key)
    log.append(run_id, 'done', cap_name, effect=effect, key=key)
    return effect


def compensate(run_id):
    # Reverse order, from the log. Never from the transcript.
    for entry in reversed(log.entries(run_id)):
        cap = REGISTRY[entry.cap_name]
        effect = entry.effect or cap.probe(entry.args)   # intent-only: go look
        if effect is None or cap.verify_undone(effect):
            continue
        cap.undo(effect)
        if not cap.verify_undone(effect):
            page_human(run_id, entry)                     # stop, don't improvise
            return
        log.append(run_id, 'compensated', entry.cap_name, key=entry.key)

The idempotency key also takes care of the improvising-retry problem. If the agent tries `create_branch` again in the same run with the same intent, the harness hands back the existing effect instead of letting it make `-2`. A retry should land on the same state, not a new one.

Step 3: Classify every call and move the irreversible ones to the end

Before the run starts, every planned call gets one of three labels:

• ▹**Reversible**: undo is cheap and complete. Branches, draft PRs, scratch files.
• ▹**Compensatable**: an undo exists, but it's semantic, not a rollback. Think migrations with a tested down, or flags with snapshots.
• ▹**Irreversible**: once it runs, someone or something outside your control has seen it. Slack posts, customer emails, paid API calls, ticket transitions that fire webhooks, anything that hits a partner's system.

Irreversible calls don't run when the agent calls them. The tool writes the request to an outbox and returns 'queued', and the agent has no way to send from the outbox. The harness flushes it only after every compensatable step has committed and the run's success condition passes (for us, CI green). Anything client-facing also waits for a human to click approve.

In saga terms this is the pivot step: past it, you stop compensating and only go forward. Most saga writeups put the pivot wherever the business flow happens to put it. With agents, put it at the end on purpose. In my experience agents fail mid-run far more often than a well-tested service does, so you want as much of the run as possible on the side of the line where undo still works.

Sometimes an irreversible step won't fit at the end. Maybe step 3 genuinely depends on a paid API response. Then that call gets a human approval gate in the middle of the run, and if the human says no, everything before it gets compensated. That's fine. The agent making that call on its own is not.

Step 4: Compensate the signals, not just the state

Moving Jira back to 'In Progress' wouldn't have saved my afternoon. QA had already started writing test cases against the flagged panel. The client had already seen 'In Review'. Restoring state reverses what the system shows. It doesn't reverse what people already did because of it.

So if a compensatable step was visible to humans for any length of time, its undo includes a correction posted where the original signal landed. That might be a comment on the ticket, a reply in the same Slack thread, or a direct note to whoever owns the QA cohort. A badly written correction just starts a second round of confusion. 'Please ignore the previous message' earns you five replies asking which message. This template has held up for us:

[Automated correction - run {run_id}]
What you may have seen: {signal}, at {time}.
Status: this was premature. It has been reverted to {previous_state}.
What you need to do: {nothing | specific action, e.g. 'discard test cases written against billing panel v1'}.
When it will come back: {after CI passes on the retry | unknown, owner will update}.
Questions: {named human owner}, not this bot.

Two rules. Post in the same place as the original signal. And always fill in 'what you need to do', even when the answer is 'nothing'. Most follow-up questions come from people who can't tell whether the mistake just created work for them.

Replaying the original run

Same task, new harness. Branch, migration and flag go through `invoke` and get logged. The Jira transition, the Slack summary and the partner API call go to the outbox. CI dies at exactly the same point as before.

Compensation took about two minutes, most of it the down-migration. The flag had been live for QA accounts for a few minutes, so the harness sent the cohort owner one correction note. That was the only message any human got. The client portal never changed. Nobody else noticed. I found out from the run report, which is how I should be finding out.

Monday checklist

• ▹List every tool your agent can call. That includes MCP servers and anything it can reach through a shell (`curl`, `psql`, CLI tools with credentials).
• ▹Next to each one, write its undo in one line. Be specific: 'delete ref', 'run down() for version N', 'restore snapshot ID'.
• ▹Write `verify_undone` for each: the query you'd run to confirm the system really is back where it was.
• ▹Mark every tool where you couldn't write the undo, or where a human or external system reads the effect. Those are irreversible.
• ▹Put irreversible tools behind an outbox that only flushes after the run succeeds, or behind a human approval.
• ▹Have the harness write an intent record with IDs before each call, to a log outside the agent's context.
• ▹Move compensation into the harness, driven by the log in reverse order. Delete 'undo what you did' from your prompts.
• ▹For every effect a human can see, write the correction message template now, before you need it.
• ▹And one rule: if you can't write the undo, the agent doesn't get the tool.

That last rule ended up being the most useful one. When we went through our own registry, a couple of tools had a blank undo column and we hadn't noticed. One of them was the tool that cost me three hours.