Measuring flow

Day 23: Measuring Flow — Why More Agents Don't Ship More Features

Five agents gave us four times the code and zero extra features. Sooner or later leadership will ask whether more agent seats will make your team faster. If you don't have a flow metric ready, you'll answer with lines of code, and lines of code will mislead you.

The dashboard moment

At PhoenixDX we went from one coding agent running in the background to five. For the next six weeks our engineering dashboard looked fantastic. Lines of code per day rose about 4x. PRs opened per day rose too. Then someone added a second chart beside it: merged features per week. That line was flat, and it stayed flat for all six weeks.

The meeting that followed opened with a fair question: "So do we buy more seats?" If 5 agents gave us 4x the code, 10 agents should give us 8x. The logic sounds right and it's wrong. This lesson explains why, and shows you how to measure it.

The wrong first guess: agent quality

We assumed first that the agents were writing bad code. More code plus more rework would explain flat output. So we pulled the rejection rate, meaning PRs closed without merging or sent back for major changes. It had barely moved. Reviewers were rejecting agent PRs about as often as before.

The code was fine. It just wasn't getting anywhere. PRs were opened and then sat untouched. So the question changed from "is the work good?" to "where is the work waiting?" One equation answers that.

Little's Law, explained with a coffee shop

Little's Law comes from queueing theory. John Little proved it in 1961. It applies to any stable system where items arrive, wait, get processed and leave, as long as you're talking about long-run averages:

Items in system  =  Arrival rate  ×  Time in system
       L         =       λ        ×        W

Rearranged the way engineering managers care about:

Throughput  =  WIP / Cycle time

Think of a coffee shop. People walk in, order at the till, wait, collect their coffee and leave. If about 10 people are inside at any time and each stays 5 minutes, the shop serves 2 customers a minute. Change one of those numbers and at least one of the others has to change with it.

• ▹WIP (work in progress): items that have started and aren't finished. In the shop, the people inside. In your team: open PRs, in-flight agent tasks, unmerged branches.
• ▹Cycle time: how long one item takes from start to done. Door to door for a customer. For you, from spec written to change running in prod.
• ▹Throughput: items finished per unit of time. Coffees per minute in the shop, merged and running changes per week for you.

Here's the trap. Raise WIP while throughput stays put, and the equation forces cycle time up. If the till is the slow part, cramming more people into the shop doesn't make more coffee. You just get a longer line.

Mapping the coffee shop onto an agentic pipeline

An agent-assisted change passes through a series of stations, and every station has a capacity:

• ▹Spec: written by humans, often batched up front, and rarely the day-to-day bottleneck.
• ▹Agent generation: close to free and runs in parallel. Each new seat adds capacity straight away.
• ▹CI: machines. Need more? Pay for more runners.
• ▹Human review: one brain per reviewer and a few focus hours a day. You can't run one person in parallel.
• ▹Merge and deploy: automated and quick on most teams.

Going from one agent to five gave the shop five times the baristas, with still only one person on the till. Every agent PR ends up at the same human merge gate, so the system can't ship faster than that gate clears work, however quick generation gets. That's the theory of constraints applied to agents: the slowest station sets the pace for everything.

What our numbers looked like

When we stopped staring at output and started looking at queues, it was obvious:

• ▹Open PRs went from about 4 to more than 20. WIP grew roughly 5x, in line with the agent count.
• ▹Median review wait stretched from hours to days. Throughput was flat and WIP was rising, so Little's Law said cycle time had to grow. It did.
• ▹PRs went stale. By the time a reviewer opened one, main had moved, and the agent had to rebase or regenerate against the new code. That cost more tokens and another CI run, and often a second review pass. We were paying for the same work twice.
• ▹LOC kept climbing because the agents never stopped producing. But that code was inventory sitting in a queue. None of it had reached users.

Here's a worked example with round numbers (illustrative, not our exact data). A team merges 4 agent PRs a week and keeps 20 open. Cycle time = WIP / throughput = 20 / 4 = 5 weeks. Add five more agents and WIP climbs to 40. The reviewer still clears 4 a week, so cycle time becomes 10 weeks. You get twice the code and no extra shipped work, and every PR now waits twice as long while main keeps moving, so more of them go stale.

A short aside from multi-agent systems research

My PhD is on multi-agent systems, and researchers there know this pattern well. Add agents and the cost of generating each unit falls, but coordination and verification costs go up, because there are more outputs to check, reconcile and integrate. At some point the verifier sets the limit, not the generators. On a software team the verifier is your reviewer. Cheaper code doesn't remove that limit. You just hit it sooner.

What actually fixed it

We didn't buy more seats. We put a limit at the gate:

• ▹A hard WIP limit at the merge gate: no more than N open agent PRs per reviewer. We started small and tuned it. Having an N and enforcing it matters more than which number you pick.
• ▹Queue-aware agents. When the queue is full, agents stop opening PRs. They pause, or switch to work that doesn't need the gate yet: splitting the next task smaller, writing tests for queued work, drafting the plan for the next spec.
• ▹Small PRs by contract. We pre-split agent tasks so each PR does one reviewable thing. Small PRs get reviewed faster, which cuts cycle time. They also go stale less, so you pay for regeneration less often.

The gate check itself is a few lines. This is the kind of script we run before dispatching a new agent task:

#!/usr/bin/env bash
# wip-gate.sh: block new agent work when the review queue is full
MAX_PER_REVIEWER=3
REVIEWERS=2
LIMIT=$((MAX_PER_REVIEWER * REVIEWERS))

OPEN=$(gh pr list --state open --label agent --json number --jq 'length')

if [ "$OPEN" -ge "$LIMIT" ]; then
  echo "WIP limit reached ($OPEN/$LIMIT). Agent: split or test queued work instead."
  exit 1
fi
echo "Queue has room ($OPEN/$LIMIT). Dispatching task."

Roughly, before and after: fewer agents were active at any one time, and more features merged each week. Review wait shrank because reviewers were seeing fewer, smaller PRs. Rebases and regenerations dropped because PRs merged before main had drifted far. Lower WIP brought cycle time down, exactly as Little's Law predicts. Throughput rose as well, because reviewers stopped spending their few focus hours on second passes over stale, oversized PRs.

The Monday diagnostic

Run this before anyone suggests buying more agent seats:

• ▹1. Count open agent PRs per human reviewer. If it's more than one person can review in a day or two, you already have a queue.
• ▹2. Compare median time-in-review with median time-in-generation. In our experience generation takes minutes to an hour, and review often takes days.
• ▹3. Check what share of PRs needed a rebase or regeneration before merging. You paid for that work twice.
• ▹4. Decide. If review wait is longer than generation time, more agents won't help. Cap WIP first.

# Median hours from PR opened to merged, for the last 50 merged agent PRs
gh pr list --state merged --label agent --limit 50 \
  --json createdAt,mergedAt \
  --jq 'map(((.mergedAt|fromdate) - (.createdAt|fromdate)) / 3600)
        | sort | .[length/2|floor]'

Replace the vanity metric

• ▹Stop reporting LOC per day, PRs opened and agent sessions run. All three go up when the queue backs up.
• ▹Start reporting merged-and-running changes per week (throughput) and cycle time from spec to prod.
• ▹Keep open PR count and LOC in flight as a warning light. If they climb while throughput stays flat, the queue is growing. Read LOC as queue depth, not as a scoreboard.

If you know the DORA metrics, none of this is new. Lead time for changes and deployment frequency were always the honest numbers. Agents just made the dishonest ones, LOC above all, much more tempting.

Where this goes next

Today you learned to treat the human review gate as the constraint and to cap WIP in front of it. A cap stops the damage but doesn't widen the gate. Widening it is the next lesson: writing specs and acceptance criteria precise enough that review means checking against a contract instead of open-ended judgement. That way each reviewer clears more work and the bar stays where it is.

Extend your knowledge

• ▹'The Goal' by Eliyahu Goldratt covers the theory of constraints. Its central idea, that the bottleneck sets system throughput, maps directly onto the human merge gate.
• ▹'The Principles of Product Development Flow' by Donald Reinertsen. Start with the chapters on queues, WIP constraints and batch size.
• ▹'Accelerate' by Forsgren, Humble and Kim, plus the yearly DORA research reports, on lead time and deployment frequency as outcome metrics.
• ▹This week, run the Monday diagnostic on your own repo with the gh CLI snippets above. Write down open agent PRs per reviewer and median time-in-review before you change anything.