Reversibility and canary releases

Day 26: Reversibility comes before rollout: why your canary may not protect you

A canary can abort itself in four minutes and turn every dashboard green while you keep emailing people who unsubscribed. Most teams pick a rollout strategy out of habit: 5% canary, watch the dashboards, roll back on alert. That habit only works when rollback actually undoes the damage. Once a change writes state, calls another company's API or lets an agent act, rolling back can leave all of the damage where it is. So before you decide how to release a change, find out whether it can be reversed at all.

The 'successful' canary

This is a composite of incidents I've seen or run myself. The change is v2.3 of a notifications service. It refactors how marketing consent is stored and adds a weekly digest email.

• ▹14:00. The pipeline runs the database migration first, as many pipelines do. It backfills a normalised `marketing_consent` column from three legacy fields. The mapping has a bug: any user who left one legacy field null ends up with `true`.
• ▹14:02. Argo Rollouts moves 5% of traffic to the v2.3 pods. The digest scheduler on those pods starts queueing emails.
• ▹14:09. An alert fires because the error rate on the canary pods jumped. A null-handling bug is throwing 500s on the preferences page.
• ▹14:13. The analysis step fails, the rollout aborts by itself and 100% of traffic is back on v2.2. Four minutes from alert to rollback, and nobody had to touch anything.
• ▹14:20. Every dashboard is green. Someone posts 'canary did its job' in the channel.
• ▹Next morning. Support tickets start coming in: 'I unsubscribed, why am I getting marketing email?' Two things went wrong. The canary pods sent digests for eleven minutes, and those emails are already in people's inboxes. Worse, v2.2 reads the same `marketing_consent` column the migration overwrote. The 'safe' old code is now emailing opted-out users at 100% of traffic.

The rollback brought the old code back. It did nothing about the data the migration changed, and nothing about the emails already sent.

Where this sits in the series

Earlier days covered where services run and who owns which data. This lesson puts the two together. Once you know what state a change touches and who owns it, you can see the rollout strategy for what it is: an architecture decision, not a pipeline setting. The 'canary: 5%' line in a Helm values file is where that decision ends up. It isn't where the decision gets made. If no architect ever asked what the change writes, the pipeline config made the call by default.

The hidden assumption: a canary makes two promises

When a team says 'we canary everything', it's making two separate promises:

• ▹Blast radius ('few people see it'). Only a small slice of traffic reaches the new version, so a bug hurts only a small slice of users.
• ▹Reversibility ('we can undo it'). When we spot a problem, putting the old version back restores the world to how it was before.

Argo Rollouts, Flagger and LaunchDarkly's guarded rollouts are very good at the first promise. They shift traffic or flag exposure, check metrics and roll back automatically. None of them can keep the second one. A traffic router can send requests back to the old pods. It can't un-run a migration, recall an email or reverse a captured payment. These tools assume reversibility, so checking it falls to the architect.

This has two consequences. First, a traffic percentage only limits damage from code paths that traffic triggers. Migrations, schedulers, consumers and batch jobs often run at full scope regardless of the split. Second, the metric that triggers a rollback (errors, latency) is usually not the one that measures the harm (wrong data, wrong recipients). In the incident above, the 500s were a lucky side signal. The bug that actually hurt people threw no errors at all.

Four kinds of state that break rollback

• ▹Schema and data migrations. A dropped column, a renamed field or a backfill that overwrote values. They run once against the whole dataset, so the canary percentage doesn't apply to them.
• ▹Outbound side effects. Emails, SMS, payment captures and webhooks to partners. Once the request crosses your boundary, someone else owns what happens next.
• ▹State other systems have cached or consumed. Events already read off Kafka, a downstream ETL that loaded the bad rows into the warehouse, or a CDN or client cache holding the new response shape. Rolling back your service leaves all of their copies as they are.
• ▹Agent actions. After a model or prompt bump, a support agent might close tickets it should have escalated, a coding agent might delete branches or an ops agent might message customers. The agent did exactly what its tools allowed, and a rollback reverses none of it.

The core tool: a reversibility classification

Before you choose a rollout pattern, put the change into one of four classes. Classify by the worst effect anywhere in the change. A PR that is 95% UI tweaks plus one backfill is an R1 change.

• ▹R0: Stateless, code only. Examples: UI copy, a pure function, a read-only endpoint, a performance refactor with no writes. What a canary buys you: nearly everything, because nothing outside the process changed and rollback is complete. Pattern: plain canary with automated analysis and abort.
• ▹R1: Internal state we own and can migrate back. Examples: schema changes, new write paths, changed serialisation formats, backfills. What a canary buys you: very little by default. The migration ignores the traffic split, and the old code may not be able to read the new state. Pattern: expand/contract, so old and new code can both run against the same data.
• ▹R2: Effects that leave our boundary through a choke point we control. Examples: events on a bus, queued emails and webhooks, downstream ETL feeds. What a canary buys you: detection, but only after the effect has already gone out. Pattern: shadow or dry-run mode plus a pausable outbox, so effects are recorded and compared before anything is dispatched.
• ▹R3: Irreversible external effects. Examples: money moved, a message a human has read, data deleted at a third party, an agent acting on production systems. What a canary buys you: fewer irreversible mistakes, but still more than zero. Pattern: human approval gates, rate-limited first actions and compensating actions designed before launch.

Look at the order: reversibility first, traffic split last. The split still helps in every class. Once you leave R0, though, it is no longer your main safety mechanism.

Pattern menu by class

R0: Plain canary. Define success metrics before you ship: error rate, p99 latency and a business metric such as checkout completion. Let the rollout controller compare the canary against the baseline and abort automatically. R0 is the only class where 'the canary passed' means roughly what people think it means.

R1: Expand/contract, also called parallel change. Split one breaking change into steps you can deploy separately. Expand: add the new column or table and leave the old one alone. Migrate: write to both and backfill the new one, with the backfill run as its own versioned job that you can re-run. Contract: drop the old structure only after every reader has moved, usually a release or two later. Until the contract step runs, rolling back just means deploying the old code, which still reads the old, untouched data.

-- Release N (expand): additive only, old code is unaffected
ALTER TABLE users ADD COLUMN marketing_consent_v2 BOOLEAN NULL;

-- Separate backfill job, idempotent, writes ONLY the new column
UPDATE users
SET marketing_consent_v2 = (legacy_opt_in IS TRUE AND legacy_unsub IS NOT TRUE)
WHERE marketing_consent_v2 IS NULL;

-- Release N+1: new code reads v2, compares against legacy in logs (dual-read)
-- Release N+2 (contract): only after mismatch count is zero and all readers moved
-- ALTER TABLE users DROP COLUMN legacy_opt_in;

R2: Shadow mode plus a pausable outbox. The new version runs its full logic but doesn't call the email provider or partner webhook directly. It writes the effects it intends to an outbox table, tagged with the release version. During the canary, the dispatcher skips rows tagged with the canary version, or only logs them. You then compare what the canary meant to do with what the stable version would have done: same recipients, same amounts, same payloads. Promoting means un-pausing dispatch. Rolling back means deleting rows nobody has seen. What makes this work is that dispatch is gated by release version, separately from traffic routing.

R3: Approval gates, rate-limited first actions and compensations. Some effects can't be held back forever, because a refund has to be issued and a customer has to get a reply. For these, send the first N real actions of a new version through human approval. After that, allow a small hourly budget that grows only if the reviewed actions were correct. Before launch, write down the compensating action for each effect: a reversing ledger entry for a payment, a correction email template, the API call that restores a deleted resource from its retention window. If you can't write the compensation, nobody should ship the change alone, and saying that out loud is part of the architecture review.

The agentic twist: for agents, the side effects are the product

In agentic software engineering, a 'release' often isn't a code deploy at all. It can be a prompt edit, a model version bump, a new tool or a reworded tool description. Any of these can change agent behaviour as much as rewriting a service, and plenty of teams push them out as a config change with no rollout at all. A model alias that quietly starts pointing at a new snapshot is also a release, and nobody approved it. Pin model versions so that changing a model is something someone decides to do.

Here's why a 5% agent canary gives you less protection than it seems to:

• ▹The split limits sessions, not actions. One agent session can make dozens of tool calls, so 5% of sessions can still mean hundreds of real writes in an hour.
• ▹The failure signal looks like success. When an agent closes a ticket it should have escalated, the tool call returns 200 and error rates and latency stay flat. The bad outcome looks exactly like the product doing its job.
• ▹Most agent effects are R2 or R3 by design. Agents are useful because they act: they send the message, merge the PR, issue the refund. The more useful the agent, the more of its effects end up somewhere a rollback can't reach.

So with agents, spend your effort on controlling what the agent can do, and less on the traffic split:

• ▹Read-only tool scoping for new versions. A canary agent gets real read tools, but its write tools are swapped for 'propose' variants that record the intended action. That's the agent version of R2 shadow mode.
• ▹Action-level kill switches. The tool gateway checks a flag for each tool (`close_ticket`, `delete_branch`, `send_customer_message`) at execution time, outside the agent. You can switch off one dangerous action in seconds without rolling back the whole agent.
• ▹Per-version action budgets. A new prompt or model version gets at most N write actions per hour until a human has reviewed a sample of them.
• ▹Evaluate actions, not responses. Run the canary and the stable version on the same inputs and compare their proposed actions. How often they disagree on write actions is the canary metric that matters.

# Tool gateway policy, enforced outside the agent runtime
agent: support-triage
release: prompt-v14+model-pinned-2026-09
rollout:
  session_share: 5%
  tools:
    read_ticket:        { mode: live }
    search_kb:          { mode: live }
    reply_to_customer:  { mode: propose }   # recorded, shown to a human, not sent
    close_ticket:       { mode: propose }
    issue_refund:       { mode: disabled }
  promote_when:
    reviewed_proposals: ">= 200"
    write_disagreement_vs_stable: "reviewed and accepted by owner"
  kill_switch:
    scope: per_tool
    owners: ["on-call-support-eng"]
    max_time_to_stop: 60s

Replaying the incident with the classification applied

Same change, same bugs. This time we classify v2.3 in the design review. The consent migration makes it R1. The digest emails make it R2, because they can't be recalled once sent but they pass through a queue we own. The worst class is R2, and that changes the rollout plan:

• ▹The migration is split into expand/contract. Release N adds `marketing_consent_v2` and backfills only that column. v2.2 keeps reading the legacy fields, which nobody has touched.
• ▹The v2.3 canary pods read the new column and log every case where it disagrees with the legacy fields. A few minutes after the backfill, the null-mapping bug shows up as a non-zero mismatch count. No email has gone out, and we didn't need a 500 to catch it.
• ▹The digest path writes to the outbox tagged `release=v2.3`, and dispatch for that tag is paused. A shadow comparison shows the canary wants to email users the stable logic would skip, so the same bug gets flagged a second way.
• ▹We roll back: drop or ignore the new column, delete the paused outbox rows, deploy v2.2. There's nothing left to clean up.
• ▹Customers affected: zero. It was the same bug with the same 5% split. The outcome changed because every effect of the change could still be undone while we were deciding whether to trust it.

Design-review checklist: five questions before you approve any rollout

• ▹What state does this change write? List every table, queue, cache, file and external resource. If the answer is 'none', the change is R0, and that's the only answer that justifies a plain canary.
• ▹Can old code read new state? If a rollback puts v-old back on top of data that v-new wrote, does v-old still behave correctly? If not, you need expand/contract before a canary is any use.
• ▹Which effects leave our boundary? Emails, webhooks, payments, events and agent tool calls. For each one, check whether it goes through a choke point you can pause.
• ▹What's the compensating action? For every R3 effect, name the specific action that corrects it and who runs it. 'We'd apologise' doesn't count.
• ▹Who can stop it in 60 seconds? Name the person, the switch and its scope. 'Roll back the deployment' is too slow and too blunt when a single agent tool is misbehaving.

Common pitfalls

• ▹Classifying a PR by its main purpose and missing the one backfill or webhook buried inside it.
• ▹Running migrations in the pre-deploy step, so they reach 100% of the data before the canary even starts.
• ▹Treating a prompt or model change as config and pushing it to every agent session at once.
• ▹Putting the kill switch inside the agent's own prompt or code path. It belongs in the tool gateway, where a confused agent can't route around it.
• ▹Running the contract step of expand/contract in the same release as the expand step. That quietly turns an R1 change back into one you can't undo.

Day 26 exercise

Take the last three changes your team shipped, and include at least one prompt, model or agent-tool change if you have one. For each, write down:

• ▹Its class (R0–R3), based on the worst effect in the change.
• ▹The rollout you actually used.
• ▹Whether that rollout would have saved you if the change had been wrong. If it was R1 or higher and you used a plain canary, mark it 'theatre': the ritual ran, but nothing would have been undone.
• ▹For each 'theatre' change, the one mechanism you should have added: expand/contract, a paused outbox, propose mode or a per-tool kill switch.

Coming up on Day 27: reversibility applies to architecture decisions as well as deployments. We'll connect today's R0–R3 classes to architecture decision records and use the same question, 'how expensive is it to undo this?', to decide how much ceremony a decision deserves.

Extend your knowledge

• ▹Read Danilo Sato's 'ParallelChange' entry on Martin Fowler's bliki. It gives a clear description of expand/migrate/contract.
• ▹Read the 'Transactional outbox' pattern on Chris Richardson's microservices.io, then sketch how you'd add a per-release dispatch pause to it.
• ▹Read the 'Canarying Releases' chapter of Google's The Site Reliability Workbook and note which of its assumptions only hold for R0 changes.
• ▹Go through the Argo Rollouts or Flagger docs on how analysis and abort work, then list the effects in your own system that those steps could never undo.