Feature flag rollouts

Day 22: The Rollout Is the Review

Your coding agents can open twenty PRs before lunch, and not one of them decides who actually sees the change. A feature flag decides that. Usually one person flips it, it takes about four minutes, and nobody reviews it. Once agents write a large share of your PRs, code stops being the scarce thing. What's scarce now is deciding which users get the change, how many of them, how you'll know it worked, and when the old path goes away. All of that gets decided in the rollout, so that's where your judgment needs to go.

The Friday moment

It's Friday afternoon. You look back over the week and count 31 PRs you approved. Agents wrote most of them. Tests passed and the diffs were clean. Then someone asks what changed for users this week, and you don't really know. The most consequential thing that happened all week took four minutes: someone moved a flag from 5% to 50%. Nobody reviewed it, there was no checklist, and only one person thought about it.

That's what this lesson is about. Review didn't go away. It moved somewhere else, and nobody told us. We kept spending careful human attention on the diff, which agents had made cheap. The real product decision, who gets this and when, went out with almost no attention at all.

Where this fits in the series

In earlier lessons you learned to define the outcome and the metric before you build. Today covers where that outcome actually gets decided once code is cheap, which is at the flag.

Reframe: a flag is a product dial, not a deploy trick

Most of us first learned feature flags as a deployment tool. You merge unfinished work safely, keep it hidden, and turn it on later. That's correct as far as it goes, but it skips the more important half. A flag decides which product a given user gets. If 5% of users see the new checkout and 95% see the old one, you are running two products at the same time.

• ▹Merging code is an engineering event, because it changes what could run.
• ▹Turning on a flag is a product event. It changes what users actually get.
• ▹Agents made merging cheap. You get more PRs, faster, and they're often good enough to merge.
• ▹So the flag is now the scarce decision, and it should get the attention we used to spend reading every line.

The failure mode we actually hit

Here's what happened to us. An agent-written change reworked how a dashboard loaded a user's default view. The diff looked fine, the tests passed, and a human approved it. It shipped behind a flag. A few days later the flag was at 100%, and nobody had decided that. It got there because "it merged" and the flag looked like leftover scaffolding.

The code didn't break. It did exactly what it was asked to do. The new default worked well for new users, but long-time users had built their workflow around the old view, and overnight they couldn't find what they used every day. Support tickets arrived before any dashboard showed a problem, because nobody had picked a metric to watch. This was a product miss. We never asked who the change was for, what would show it worked, or what would make us turn it off. A more careful code review wouldn't have caught it, because none of those questions were in the code.

What we changed: review the rollout, not just the diff

We moved review effort from the diff to the rollout plan. Every flagged change now ships with a short rollout card, whether a human or an agent wrote the code. Agents can draft the card from the ticket and the PR. Humans still read the code, but approving the card is now their main job. When a reviewer approves the card, they're saying three things: I know who this is for, I know what success looks like, and I know what we'll do if it goes wrong.

The rollout card (copy this)

# rollout-card.yaml — one per flag, lives next to the PR or in the flag's description
flag: dashboard_new_default_view
what_changes_for_users: >
  New users land on a simplified default view instead of the full table.
segment:            # WHO is this for? Be specific.
  include: accounts created in the last 30 days
  exclude: users with saved custom views
exposure_steps:     # HOW MANY, in order. Each step has a minimum soak time.
  - internal staff        # min 2 days
  - 5% of segment         # min 1 week
  - 50% of segment        # min 1 week
  - 100% of segment
success_metric:     # ONE metric that proves it worked
  name: day-7 return rate for new accounts
  expected: goes up vs control
guardrail_metric:   # ONE metric that must NOT get worse
  name: support tickets tagged 'dashboard' per 1k active users
rollback_when: >
  Guardrail is clearly worse than control at any step, or any data-loss report.
  Rollback = set flag to 0%, no deploy needed.
owner: <name>       # a human who answers for this decision
kill_date: 2026-12-15   # flag AND the losing code path are deleted by this date
flag_changes_by: humans only (agents may propose, not flip)

• ▹Segment: a real group of users. Don't default to "everyone." If you can't name the segment, you haven't decided who the change is for.
• ▹Exposure steps: internal → 5% → 50% → 100%, with a minimum soak time at each step. Someone has to choose to move to the next step. A calendar reminder isn't a decision.
• ▹One success metric and one guardrail. The first shows the change worked. The second shows you didn't break something next to it. Add more and nobody will look at any of them.
• ▹Rollback condition: write it down before launch, while you're calm and before you've gotten attached to the result.
• ▹Owner: one named person. A team or a Slack channel doesn't count.
• ▹Kill date: the date the flag and the losing code path get deleted.

Why the kill date matters more with agents

A stale flag leaves dead branches in your codebase: the old path, the new path, and an if-statement that has been true for six months. Humans learn to skip past them. Coding agents often don't. In our experience they tend to treat every branch as live behavior. They change both paths to keep them consistent, write tests for code no user will ever hit, and sometimes "fix" the dead branch in ways that confuse the next reviewer. Deleting the flag is what completes the product decision, because it says this is now the product. It also leaves agents a cleaner context to work in. The good news is that agents are very good at writing the removal PR once the metric has held. Put the kill date on the card, and when that date arrives, have an agent open the cleanup PR.

What this asks of you

This is where the product mindset comes in. You can't honestly approve a rollout card unless you know who the user is, which metric matters, and what it costs to be wrong. Approving a diff takes engineering skill. Approving a rollout takes product judgment. Days 1–21 stop being theory here. Personas, outcomes over output, picking a metric and cost of delay all end up in one short card that you sign.

• ▹Can't name the segment? Go back to the user and the problem.
• ▹If you can't pick one success metric, the outcome was never defined in the first place.
• ▹No rollback condition means you don't yet know what failure looks like.
• ▹And if nobody can set a kill date, nobody has decided what "done" means.

Honest limits

• ▹Not every change needs a flag. Copy fixes, internal refactors that don't change behavior, and clear bug fixes can ship directly. Use a flag when there's a real product choice or real risk.
• ▹Flags add complexity. Each live flag can double the number of path combinations you have to test and reason about, and flags that interact with each other make it worse. That's one more reason to set kill dates.
• ▹Flipping a flag is a release, so it needs its own review and an audit trail: who changed it, when, and why. Plenty of teams log deploys carefully and flag changes not at all. If that's your team, fix it first.
• ▹Be strict about agents and flag state. An agent that can change production flag percentages is shipping product to users. Let agents propose the change and draft the card, and keep the actual flip behind human approval.

Today's exercise

Pick one feature your team has behind a flag right now and write its rollout card using the template above. Give it about 15 minutes. If you get stuck on the success metric or the kill date, that tells you something useful: there's a product decision nobody has made yet. Take the half-filled card to your PM or tech lead and make that decision together.

Next up: Day 23

Once you're rolling out in steps, you need to read what the 5% is telling you. Tomorrow covers how to read experiment results without fooling yourself.

Extend your knowledge

• ▹Pete Hodgson's "Feature Toggles (aka Feature Flags)" on martinfowler.com explains the categories of flags (release, experiment, ops, permission) and why each one needs a different lifespan.
• ▹Look at OpenFeature, the CNCF's vendor-neutral flag API. While you're there, check whether your own flag tool (LaunchDarkly, Unleash, Flagsmith or homegrown) logs who changed flag state and why.
• ▹Uber's Piranha is an open-source tool that automatically removes stale feature flags. Compare it with having a coding agent write the removal PR on the kill date.
• ▹To prepare for Day 23, read "Trustworthy Online Controlled Experiments" by Kohavi, Tang and Xu, especially the chapters on guardrail metrics and common analysis pitfalls.