Git Revert Isn't a Rollback — It's a 40-Minute Bet You're Making at 2AM

Day 17 — Feature Flags: The Rollback Mechanism That Actually Works

Every postmortem template has the same line: "rollback initiated at [time]." Almost everyone fills that in with a git revert. That one habit is the reason 2am incidents that should end in seconds instead turn into 40-minute fire drills — because a revert only fixes the repo. It does nothing about the thing that's actually on fire.

The 40 minutes that shouldn't have happened

2am. A new ranking model goes live and starts throwing 500s downstream. The on-call engineer does exactly what the runbook says: git revert, push, wait for CI, rebuild the image, redeploy, wait for the rollout to crawl across every pod, re-verify. Forty minutes end to end, error rate elevated the entire time. Meanwhile, in the incident channel next door, another team hits a bad release of their own — same night, same severity — and closes it in about ten seconds. No revert. They flipped a flag. The code that caused the mess never left prod; it just got switched back off. Same postmortem template, same question — "when was rollback initiated?" — two wildly different answers: one forty minutes after detection, one ten seconds after. Same severity, same template. The gap wasn't who typed git commands faster. It was what each team had built before the incident ever started.

The one sentence to leave with

A feature flag splits deploy from release. Deploy is code landing on a server. Release is that code actually running for a real user. The moment you separate those two events, "rollback" stops meaning "undo a commit" and starts meaning "flip a boolean." That's the whole trick. Everything else in this lesson is just that one sentence played out to its logical end.

Why git revert is theater once you've deployed

Before code reaches prod, git revert is the real thing — it stops a change before it ever touches reality. After code reaches prod, it's theater. It looks like undoing the change, but you've actually just started a new, slower change moving in the opposite direction. In the gap between "bad code went live" and "revert finally lands," reality has kept moving without you:

• ▹Caches are already holding values the new code computed — reverting the code doesn't touch them.
• ▹Migrations may have already run against the database — a commit going backward doesn't take the schema with it.
• ▹Requests already in flight are executing against the new code right now — they finish on whatever logic was live when they started, revert or no revert.
• ▹Downstream consumers — other services, webhooks, an agent that just read a tool's new output schema — have already reacted to the new behavior, sometimes by persisting a decision based on it.
• ▹If you're running AI specifically: a bad model version, a changed system prompt, a new tool grant — reverting the code doesn't un-run the requests the model already answered, or undo the action an agent already took with a tool it should never have had.

git revert answers one question: what does the repository look like now? It has nothing to say about what's currently live, or how fast you can make it stop being live. Only a flag answers that, because the flag sits at runtime, gating execution directly. A revert has to survive CI, a build, and a rollout before it even gets a shot at touching runtime.

The reframe: containment first, experimentation second

Most engineers meet feature flags through A/B testing tools — LaunchDarkly, Unleash, some homegrown config table — and quietly file them under "experimentation infrastructure." That's the wrong mental model, and it's exactly why flags get skipped on "simple" changes — which are, of course, the changes that end up paging someone at 2am. A flag's real job is blast-radius containment: a control surface that turns a bad change off without waiting on a deploy cycle. A/B testing, gradual rollout, targeting by segment — those come free once that control surface exists. They are not why you build it. This matters even more with AI in the mix. A new model version, a changed prompt, a wider tool permission for an agent, a new retrieval index — none of that is "just code," and none of it is something you can unit-test your way into trusting. You can't prove a new system prompt won't push an agent into a destructive edge case. What you can do is ship it dark, expose it to a thin slice of traffic, and cap the damage to that slice when you're wrong. Flags are how you buy the right to be wrong safely — true whether you're shipping a checkout button or a fleet of autonomous agents.

The self-check: read your own runbook

You don't need a survey for this. Pull up your last incident runbook or postmortem and look at the mitigation step:

• ▹If it says "roll back the deploy" or "revert the commit," the code that caused the incident almost certainly wasn't behind a flag — wasn't decoupled from deploy — and your mitigation time is bounded by your CI pipeline, not by how fast a human can react.
• ▹If it says "flip flag X to off" or "drop rollout to 0%," the control surface already exists, and mitigation time is bounded by detection, not by deploy.
• ▹For AI-heavy teams, a sharper gut check: are your model version, your system prompt, and your agent's tool grants each behind their own independent flag — or are they all baked into the same deploy as the application code? Ship a new model version in the same deploy as a database migration and you can't turn one off without dragging the other down with it.

Where this goes next

Once a flag exists purely as a rollback lever, the next question answers itself: what else can you hang off that same control surface? Targeting, staged rollout, a kill switch wired to a metric instead of a human pulling the trigger. Tomorrow picks up right there — flags stop being a single on/off switch and start turning into a control surface you program against.

Extend your knowledge

• ▹Read Pete Hodgson's "Feature Toggles" piece on martinfowler.com — it's the widely cited deploy-vs-release framing, written before the AI wave, and it maps onto agent rollouts almost without changing a word.
• ▹Look at how LaunchDarkly and Unleash document "kill switch" patterns specifically. Most vendor docs lead with experimentation — skip past that to the operational sections.
• ▹Audit one system you actually own: find a change from the last month that shipped without a flag. Ask yourself what the mitigation path would have looked like if it had paged someone at 2am.
• ▹If you're running any agentic pipeline, check whether model version, prompt version, and tool grants can each be flipped independently. This is the single highest-leverage flag audit an AI-era team can run.