Your Deploy Frequency Graph Went Vertical — That's the Warning Sign, Not the Win

Day 2: Deploy Frequency Is Lying to You Now

Your deploy-frequency graph is about to go vertical, and that's the problem. Once agents start opening and merging their own PRs, that number climbs no matter what you do — which means it stops measuring anything. The metric that should have you worried right now is the one most teams quietly stopped watching closely.

The dashboard screenshot

A few weeks after we let agents at PhoenixDX open and merge their own PRs on low-risk paths, our deploy-frequency graph went vertical. Someone on leadership screenshotted it into Slack with one line: 'so are we shipping faster now?' The honest answer wasn't yes or no. It was: that graph doesn't mean what it used to mean anymore.

Why the question was harder than it should've been

DORA's four keys — deploy frequency, lead time for changes, change failure rate, time to restore — were built around one assumption: a human is the bottleneck between code written and code deployed. Deploy frequency was a good proxy for team health for exactly that reason. Shipping more often meant a team had beaten the bottleneck — smaller batches, less manual gatekeeping, tighter feedback loops. It was never the goal itself. It was a stand-in for 'we removed friction from a human-limited process.'

Agentic coding tools don't remove friction from that process — they remove the bottleneck it was measuring. An agent doesn't get tired, doesn't context-switch, doesn't wait for stand-up to decide what to work on next. Take away the constraint the metric was built to detect, and the metric keeps producing numbers. It just stops pointing at anything real.

What changed concretely

At PhoenixDX, deploy frequency stopped correlating with customer value or team throughput. It started correlating with something dumber: how many small commits an agent happened to chunk its work into. An agent that splits one feature into eight tiny PRs looks like 8x the 'shipping velocity' of a human who ships the same feature in one. Same work, same value delivered, wildly different number on the dashboard. That's not a health signal. That's an artifact of how granular the agent's planning happened to be.

• ▹What deploy frequency was built to measure: friction removed from a human-limited release process
• ▹What it measures now: how an agent decided to chunk its own commits
• ▹Those aren't the same thing — and treating them as the same thing is how you end up optimizing for noise

The part nobody wants to say out loud

Here's the uncomfortable finding: change failure rate was the metric that should have been climbing in alarm, and we nearly missed it. As PR count went up, review rigor per PR quietly went down — not because anyone chose to review less carefully, but because the sheer volume made deep review of every single PR practically impossible. We were flying on degraded signal for the one metric that actually protects users, while celebrating a vanity number that was climbing for reasons that had nothing to do with quality.

That's the trap. Deploy frequency is loud and easy to screenshot. Change failure rate creep is quiet, lagging, and usually only surfaces once an incident review connects the dots after the fact. If leadership is watching the loud metric and nobody's rigorously watching the quiet one, you'll find out you had a problem the same way your customers do — after it happens.

The reframe

In an agentic pipeline, deploy frequency gets demoted — from primary metric to denominator. Stop reporting it on its own; start using it as the unit you divide by. What matters isn't change failure rate per deploy. It's change failure rate per unit of review attention. If review attention per change keeps dropping while deploy count keeps rising, your failure rate is a ticking clock, whether or not it's spiked yet.

What we changed at PhoenixDX

• ▹Stopped reporting raw deploy count to leadership as a headline number — it now shows up as context, never as a KPI
• ▹Started reporting change failure rate as a trend line, not a single snapshot, so a slow creep is visible before it becomes an incident
• ▹Added a paired metric: 'PRs merged without a human catching a substantive issue' — a proxy for review rigor eroding under volume
• ▹Any week where PR count rises and that paired metric also rises gets flagged, no matter how good the deploy graph looks

Close

If deploy frequency is dead as a north star for agent-paced teams, the next question is obvious: what replaces lead time for changes, since it's built on the exact same broken assumption? That's Day 3 — redefining lead time once 'writing code' stops being the slow part.

Extend your knowledge

• ▹Re-read the original DORA/Accelerate research (Forsgren, Humble, Kim) with an eye for which assumptions were baked in about who — or what — writes the code
• ▹Audit your own dashboards: is deploy frequency reported as a standalone KPI anywhere leadership sees it? If so, pair it with a review-rigor metric before the next report
• ▹Track 'PRs merged without a human catching a substantive issue' for two weeks on your team and see if it moves in the direction you'd expect
• ▹Look at your CI/CD pipeline's PR-size distribution since introducing agentic tooling — a shrinking median PR size with a rising deploy count is the exact artifact this lesson describes