Blue-green deployment with state

Day 25: Blue-green fails at the database, not at the load balancer

Blue-green promises a two-second rollback, and for compute it delivers. That stops being true once your new version writes data the old version can't read. The router still flips back, but now it sends your traffic to a second broken system. Most teams learn this during an incident.

Cold open: the two-second flip

It's a startup, Thursday afternoon, and a routine release. Green is up and its health checks pass. Blue is serving traffic as usual. We flip the router, which takes two seconds. Then errors on the orders service spike on the dashboard. Someone in the channel types 'roll back'. That's a reasonable call, and it's exactly what blue-green is for. We flip back to blue, which also takes two seconds.

Then blue starts failing on every order row it reads: 'validation error: field amount required'. Nothing about blue had changed. It was running the same code as yesterday. What had changed was the shape of the rows it was reading.

Rewind 60 seconds: what blue-green promises

You run two identical production stacks, blue (current) and green (new). One router or load balancer points at one of them. To release, you deploy to the idle stack, test it and swap the pointer. To roll back, you swap it back. Think of it as two machines behind one switch, both connected to the same database.

Look at where the arrows end. The slide shows two of everything except the component that matters most. There is only one database, and both colors use it.

What actually happened, minute by minute

The ticket said 'store order amounts as integer cents, not decimal dollars'. That's a sensible change, because floating-point money bugs are real. A coding agent wrote the PR: the migration, the model change and the updated queries, all in one diff. Tests passed and a human approved it. Each piece was correct on its own. They broke because they shipped together.

-- 0042_orders_amount_cents.sql  (shipped in the SAME release as the code that needs it)
ALTER TABLE orders RENAME COLUMN amount TO amount_cents;
ALTER TABLE orders ALTER COLUMN amount_cents TYPE bigint
  USING (amount_cents * 100)::bigint;   -- full table rewrite, ACCESS EXCLUSIVE lock

• ▹T+0:00. Green deploys. Its health check is a shallow /healthz that never touches the orders table, so it passes.
• ▹T+0:02. The router flips to green. Our pipeline ran migrations as a post-switch step, so green goes live against the old schema.
• ▹T+0:05. Green queries amount_cents, which doesn't exist yet, and errors spike. The migration job starts and takes an exclusive lock to rewrite the table. Blue's last in-flight connections and all of green's requests queue up behind that lock.
• ▹T+~1:00. The migration commits, and nobody notices because everyone is watching the error graph. This is the moment rollback stopped being possible: the old column is gone and the data has a new shape.
• ▹T+~1:10. Green's errors drop off because its schema now exists. It starts inserting orders that only have amount_cents.
• ▹T+~1:30. The 'roll back' call made at T+0:05 is finally carried out. The router flips to blue. Blue's repository does SELECT * into a strict model that requires amount, so every row fails validation.

The rollback window didn't close when we flipped the router. It closed on a COMMIT that no dashboard showed. Flipping back didn't restore the old system either. It put a second version of the app in front of data it couldn't read.

The core idea: compute is reversible, state is one-way

Compute is stateless. You can replace a container, a pod or an inference server running new model weights, then swap it back, and nothing is lost. State doesn't work that way. Every write in a new shape, every dropped column and every message published in a new format moves the system forward, and swapping the router can't undo any of it.

Your real rollback window runs from the flip to the first incompatible change to shared state. That change might be a destructive migration, a write N-1 can't read or a message N-1 can't consume. If migrations run before the flip, the window can be negative: blue breaks before green has served a single request. On most teams I've worked with, nobody measures this window. People assume it's however long the router takes, which is about two seconds.

Coding agents make this worse in a specific way. An agent working on one ticket optimises for 'this PR works', and its tests run one version against one schema. N-1 compatibility is a property of two versions together, and no single-PR test suite checks it unless you build one on purpose. As agents ship more migrations, faster, the windows get shorter, and nobody is watching them.

Expand/contract, walked through on our column rename

Expand/contract, also called parallel change, splits one breaking change into several releases that are each backward-compatible. The rule is that at every step, every version that might be running can safely read and write the shared tables. Here's the amount to amount_cents change done properly.

Step 1, expand. Add the new column while blue keeps running. Ship it on its own, with no application change. Watch for one trap here. Our blue used SELECT * into a strict model that rejected unknown fields, so even adding a column would have broken it. Expand only works if readers tolerate columns they don't recognise, so select columns explicitly.

-- Release A (expand): ships alone. Blue v1 is untouched.
ALTER TABLE orders ADD COLUMN amount_cents bigint;  -- nullable, no default: cheap metadata change

-- Keep both columns consistent no matter which color writes.
-- The UPDATE branch is the part people (and agents) usually get wrong.
CREATE OR REPLACE FUNCTION orders_sync_amount() RETURNS trigger AS $$
BEGIN
  IF TG_OP = 'INSERT' THEN
    IF NEW.amount_cents IS NULL AND NEW.amount IS NOT NULL THEN
      NEW.amount_cents := round(NEW.amount * 100);
    ELSIF NEW.amount IS NULL AND NEW.amount_cents IS NOT NULL THEN
      NEW.amount := NEW.amount_cents / 100.0;
    END IF;
  ELSE
    IF NEW.amount IS DISTINCT FROM OLD.amount
       AND NEW.amount_cents IS NOT DISTINCT FROM OLD.amount_cents THEN
      NEW.amount_cents := round(NEW.amount * 100);      -- blue updated the old column
    ELSIF NEW.amount_cents IS DISTINCT FROM OLD.amount_cents
       AND NEW.amount IS NOT DISTINCT FROM OLD.amount THEN
      NEW.amount := NEW.amount_cents / 100.0;           -- green updated the new column
    END IF;
  END IF;
  RETURN NEW;
END $$ LANGUAGE plpgsql;

CREATE TRIGGER orders_sync_amount
  BEFORE INSERT OR UPDATE ON orders
  FOR EACH ROW EXECUTE FUNCTION orders_sync_amount();

Step 2, dual-write and backfill. Deploy green (Release B). It writes both columns, reads the new one first and falls back to the old one. Backfill historical rows in small batches so locks stay short. Blue serves traffic throughout and is unaffected.

# Release B (green): dual-write, tolerant read
def save_order(db, order):
    db.execute(
        "INSERT INTO orders (id, customer_id, amount, amount_cents) VALUES (%s, %s, %s, %s)",
        (order.id, order.customer_id, Decimal(order.amount_cents) / 100, order.amount_cents),
    )

def amount_cents_of(row):
    # Rows written by blue, or not yet backfilled, may only have the old column.
    if row["amount_cents"] is not None:
        return row["amount_cents"]
    return int(round(row["amount"] * 100))

# Backfill job: repeat until it updates 0 rows
BACKFILL = """
UPDATE orders SET amount_cents = round(amount * 100)
WHERE id IN (SELECT id FROM orders WHERE amount_cents IS NULL AND amount IS NOT NULL LIMIT 5000)
"""

Step 3, flip traffic. Both columns are populated and kept in sync, so this step should be uneventful. The flip itself doesn't change any state.

Step 4, soak. Blue stays deployable and both columns stay maintained. This is the only period where flipping back is a genuine rollback, because either color can read every row. Agree on exit criteria before you start: the error budget has held; the backfill is verified, meaning a count of rows where amount_cents IS NULL returns zero; your slowest scheduled jobs (month-end reports, weekly reconciliations) have run at least once on green; and nobody has had to flip back. Plan the soak in days, not minutes.

Step 5, contract. This is a separate release, days later, with a named owner. First, ship a code change that stops reading and writing amount (Release C). Then, once no deployed version references the column, including the idle blue stack, drop it (Release D). After that, your rollback target is Release C, not blue.

-- Release D (contract): separate deploy, after soak exit criteria are met.
-- Precondition: no running or idle version reads or writes 'amount'.
DROP TRIGGER orders_sync_amount ON orders;
ALTER TABLE orders DROP COLUMN amount;

Our original PR was one release. Done safely, it takes four: A, B, C and D. That's what blue-green really costs. The router config is the cheap part.

The other stateful traps the router diagram hides

• ▹Queues with in-flight messages. Green publishes v2 events, and after a flip back blue's consumers fail on them. The reverse also happens, with blue's leftover v1 messages reaching green's consumers. Messages need expand/contract too: consumers learn to read v2 first, and producers start emitting v2 later.
• ▹Caches holding serialized objects. Green writes a new shape to Redis under the same key. Blue deserializes it and either crashes or silently drops fields. Put a schema version in the key or the payload.
• ▹Sessions. If the cookie or session-store format changes, every flip logs users out or, worse, leaves them with half-parsed sessions.
• ▹Webhooks and third-party callbacks. A payment provider or OAuth redirect may be registered against one color's endpoint, or carry state (a nonce, a signature version) that only one color understands. In our incident, callbacks reached a stack that couldn't process them.
• ▹Embeddings and vector indexes. A new embedding model on green produces vectors in a different space, often with different dimensions. If the dimensions differ, blue's queries against the shared index fail outright. If they happen to match, you have a worse problem: blue gets nonsense results and no errors. Build a new index alongside the old one. That's expand/contract again.
• ▹Durable agent runs. Long-running agent workflows checkpoint their state in Temporal, LangGraph checkpointers or a table of your own. A run started on green and resumed on blue replays against code that doesn't match the checkpoint. This is why Temporal has explicit workflow versioning.
• ▹LLM response and prompt caches, plus conversation memory. Green's new prompt template writes memory or cached responses in a structure that blue's agent later reads back as context. There are no errors, only answers that are quietly worse.
• ▹Agent tool contracts. When green changes the schema of its MCP tools or internal APIs, agents built against the old schema keep calling the old shape. Treat tool schemas as public APIs and version them.

# Tolerant reader for events: works whichever color produced the message
def decode_order_event(raw: dict) -> OrderEvent:
    v = raw.get("schema_version", 1)
    if v == 1:
        cents = int(round(Decimal(raw["amount"]) * 100))
    elif v == 2:
        cents = raw["amount_cents"]
    else:
        raise UnknownSchemaVersion(v)   # route to dead-letter queue, don't crash the consumer
    return OrderEvent(id=raw["id"], amount_cents=cents)

The architect's lens

In a design review, don't ask 'do we have blue-green?' Ask 'what is our rollback window, and what ends it?' A good answer names a specific event, such as 'the contract migration in Release D, owned by Linh, scheduled after a seven-day soak.' If the answer is 'we can always flip back', the team hasn't looked at its state.

This connects two earlier parts of the series. On the data modelling day we treated the schema as a contract, and during a cutover two versions of your code depend on that contract at the same time. On the resilience days we treated recovery as something you design and measure, like RTO. A rollback you've never measured is like a backup you've never restored. On an AI-first team, write the N-1 rule down where your agents will read it (CLAUDE.md, AGENTS.md, PR templates) and enforce it in CI. Don't rely on a reviewer catching it in a 600-line diff.

# Sketch: run the PREVIOUS release's tests against THIS PR's schema
# (assumes every production release is tagged)
n-minus-1-compat:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
      with: { fetch-depth: 0 }
    - name: Apply this PR's migrations to a fresh database
      run: ./scripts/migrate up
    - name: Swap in the code from the last release tag
      run: git checkout "$(git describe --tags --abbrev=0)" -- src/ tests/
    - name: N-1 integration tests against the new schema
      run: pytest tests/integration

Cutover checklist for your next review

• ▹Is every migration backward-compatible with N-1? Prove it with a CI job that runs the previous release against the new schema. A reviewer's opinion isn't enough.
• ▹Do destructive changes (drop, rename, type change, NOT NULL on an existing column) ship in a separate, later release than the code that stops using them?
• ▹Do serialization formats (events, cache entries, sessions, agent checkpoints, embeddings) carry an explicit schema version, and can readers handle both old and new?
• ▹Does the contract step have a named owner and a date? Without both, it either never happens or happens too early.
• ▹Have you rehearsed the rollback with real writes? Flip to green, let it write, flip back and confirm blue can read what green wrote. Flipping back on an empty staging environment proves nothing.
• ▹Do health checks exercise the data path, or do they just return 200 from /healthz?
• ▹Do migrations run before the flip, after it or as a separate step? Write down which, and what that does to the window.

Back to the incident: forward-fix, not rollback

We recovered by going forward. Once the migration had committed, green was the only version that could read the data correctly. Its earlier errors came from the gap between the flip and the migration, not from bad code. So we flipped back to green and froze the router. Then we reconciled everything that happened during the bounce: orders blue had failed to save, and webhook callbacks that reached blue while it was broken, which we replayed from the provider's dashboard. A rollback would have meant writing a reverse migration against a live table under pressure, which is how a bad afternoon ends in lost data.

The rule we adopted afterwards fits on a sticky note: no migration ships in the same release as the code that needs it. Schema changes go first, on their own, and the code follows. Destructive changes go last, as a separate deploy with an owner. Every PR, whether a human or an agent wrote it, gets checked against this rule.

Expand/contract makes releases safe for state, but it also exposes another coupling: deploying code and releasing a feature were the same event. Day 26 separates them with feature flags and progressive delivery, so green's new behaviour can sit switched off in production until you decide to turn it on.

Extend your knowledge

• ▹Read Martin Fowler's bliki entries 'BlueGreenDeployment' and 'ParallelChange' (the second by Danilo Sato) one after the other. The second covers what the first leaves out.
• ▹Read 'Refactoring Databases' by Scott Ambler and Pramod Sadalage. It catalogues database refactorings, each with its own transition period, so it's essentially expand/contract worked out in full detail.
• ▹Try pgroll on a staging copy of a Postgres database. It automates expand/contract by serving the old and new schema versions at the same time. On MySQL, gh-ost handles a different part of the problem: it runs schema changes online without long table locks, but you still have to sequence the expand and contract steps yourself.
• ▹Exercise: take your last three releases that touched the database and write down the exact event that ended each one's rollback window. If you can't name the event for one of them, start your work there.