Backpressure

Day 23: Backpressure, and why your slowest consumer is a human

A pipeline runs at the speed of its slowest stage. If you never decide what happens when that stage fills up, the system makes the call for you. In agentic engineering it nearly always makes the same call: it quietly lets quality slide.

The 41-PR Thursday

Here's the setup. Twenty coding agents are chewing through a backlog. By Thursday afternoon there are 41 open agent PRs, and CI is green on most of them. One reviewer is getting through them at roughly 90 seconds each. Merged PRs per week are flat, the same number we had when we ran far fewer agents. Two weeks later a regression gets traced to a PR merged that Thursday, and nobody remembers approving it.

The agents did what they were told. CI ran. The reviewer went as fast as they could. So what broke?

Name it: backpressure

Backpressure is how a consumer that's at capacity gets its producers to slow down, so work doesn't pile up where nobody can see it.

Think of a restaurant. When the kitchen is full, it stops taking orders. It doesn't keep accepting tickets and then burn every third plate. The host at the door says "20 minutes", and the people waiting decide for themselves: stay, go to the bar, or leave. The kitchen's limit shows up at the front of the house.

The earlier lesson on queues and async showed you that a queue separates a producer from a consumer in time. Backpressure covers what that leaves out. Nothing separates them in capacity, so the consumer's limit will surface somewhere eventually. With backpressure, you get to pick where.

The architect's move: find the slowest consumer

Draw the pipeline and put a rough service rate on every stage. You don't need precision. You need the order of magnitude. Ours looked roughly like this:

• ▹Planner: one LLM call that splits a task in seconds. Apart from API rate limits, there's no real ceiling.
• ▹Agents: each one is slow, but adding more is a config change. Twenty of them turn out a lot of PRs per hour.
• ▹CI: a few minutes per run, and it scales horizontally if you're willing to pay for runners.
• ▹Human review: a careful review of a non-trivial agent PR takes most of an hour. That means reading the diff, checking it against the spec and sometimes running it locally. Around 6 a day per reviewer is realistic, and no config change will scale it.
• ▹Merge: instant.

Line the stages up and human review is at least ten times slower than anything else. Nobody had modelled that stage. We had dashboards for token spend, agent success rate and CI duration. We had nothing for review capacity, because nobody saw the reviewer as part of the system. But the reviewer is a component like any other: a service rate, a queue and failure modes.

This is Goldratt's Theory of Constraints from The Goal, applied to software. Your bottleneck sets the throughput of the whole system. Add capacity anywhere else and you get more inventory, not more output. With agents, that inventory is open PRs.

Why unbounded queues fail quietly

Little's Law connects three numbers for any stable queue: average items in the system = arrival rate × average time each item spends in the system. Rearrange it and you get the form you'll use day to day:

wait time ≈ queue length ÷ throughput

queue length   = 41 open agent PRs
throughput     ≈ 6 careful reviews / day (one reviewer)

wait time      ≈ 41 ÷ 6 ≈ 7 working days

# and that's if the agents stop opening PRs today, which they won't

So a PR opened on Thursday gets its careful review about a week and a half later. In that time main has moved on, the PR needs a rebase, the agent's context has evaporated, and the reviewer ends up reading a stale diff.

People don't throw an error when this happens. They adapt, usually in roughly this order:

• ▹They skim instead of reading. "CI is green and the description looks right."
• ▹They trust the agent's summary over the diff.
• ▹They approve in batches to get the count down.
• ▹They stop requesting changes, because every revision cycle puts another item in the queue.

When a message broker's consumer is overloaded, it falls behind or drops messages, and a metric tells you: lag, queue depth, drop count. A human consumer keeps reporting success and just does less per item. Rubber-stamping is how dropped messages show up in human systems. It's worse than a drop, because the approval says "reviewed" about a PR nobody really reviewed.

That's what broke on Thursday. No single component failed. An unbounded queue sat in front of the slowest consumer, and that consumer quietly gave up quality to keep up throughput.

The three parts of any backpressure design

• ▹1. A bound. A hard limit on how much work can wait in front of the consumer. Without one, all you have is a buffer that keeps growing until something else breaks.
• ▹2. A signal that goes upstream. The producer has to learn the bound has been hit, ideally before it does more work rather than after.
• ▹3. A defined response from the producer. Wait, retry later, shed load, degrade, or something else, but decided up front and not improvised.

Drop any one of these and you get a predictable failure. With a bound and no signal, you're dropping work silently. With a signal and no defined response, you get alert noise. With a response and no bound, the response never fires.

You've already met all three parts in systems you use:

• ▹TCP receive window. The receiver advertises how much buffer it has left, which is both the bound and the signal. When the window hits zero, the sender stops (the response). It's part of the protocol.
• ▹Kafka consumer lag. Kafka is pull-based, so consumers read at their own pace and lag builds up. Lag is a signal. It isn't backpressure by itself. You still have to connect it to a response, like scaling consumers or throttling producers. Plenty of teams watch lag and never decide what happens when it grows.
• ▹Erlang/Elixir mailboxes. They're unbounded by default, so a slow process can quietly eat memory until the node goes down. That's why demand-driven libraries like GenStage exist: the consumer explicitly asks for N items. Worth remembering that the default is often no backpressure at all.
• ▹HTTP 429 Too Many Requests, ideally with Retry-After. The server owns the bound, the status code is the signal, and the client is supposed to back off. LLM provider APIs send you these every day, so your agent harness needs an actual policy for them, not just a retry loop.

What we changed

We capped it at 3 open agent PRs per reviewer, and the orchestrator enforces the cap. That's what matters. It's not a guideline, and it's not a line in the agent's prompt asking it nicely not to open too many PRs. A prompt is a request. A gate in the orchestrator is a control. The agent can't open PR number four because the orchestrator never offers it that action.

MAX_OPEN_PER_REVIEWER = 3

def next_action(agent, task):
    reviewer = assign_reviewer(task)
    open_prs = count_open_agent_prs(reviewer)

    if open_prs < MAX_OPEN_PER_REVIEWER:
        return OpenPR(task, reviewer)

    # Bound hit: emit a signal someone can see, then switch modes.
    metrics.incr("review_gate.blocked", tags={"reviewer": reviewer})
    return pick_review_cost_reducer(agent, reviewer)

def pick_review_cost_reducer(agent, reviewer):
    # Blocked doesn't mean idle. It means: make the queue cheaper to drain.
    for pr in open_agent_prs(reviewer):
        if pr.diff_lines > 400:
            return SplitDiff(pr)
        if not pr.has_tests_for_changed_paths:
            return AddTests(pr)
        if not pr.has_verification_note:
            return WriteVerificationNote(pr)  # spec criteria -> evidence
    return Idle(reason="no review-cost work available")

Blocked mode is where this pays off with agents. A blocked TCP sender can only wait. A blocked agent is nearly free to repurpose, so we send it to work on the queue that's blocking it:

• ▹Split large diffs. A 900-line PR becomes three a reviewer can actually keep in their head. Split PRs still count toward the cap, so splitting can't be used to get around the bound.
• ▹Add tests for the changed paths. Now the reviewer checks that the tests describe the right behaviour, instead of tracing every branch by hand.
• ▹Write a self-verification note against the spec. For each acceptance criterion it says where it's implemented, how it was tested, and what the agent couldn't verify. The reviewer starts with the gaps instead of at line 1.

Notice what this response does. It doesn't only slow producers down. It also raises the consumer's effective service rate. A smaller PR with tests and a verification note takes less time to review with the same care. That's backpressure in its agentic form: blocked producers spend their time making the consumer faster.

Results, stated honestly

• ▹Fewer PRs opened. We expected that. The cap is a ceiling.
• ▹More PRs merged per week. Shorter waits meant fewer stale rebases, fewer abandoned PRs and fewer revision loops on outdated code.
• ▹Review time per PR went back up, which is what we wanted. The 90-second approvals were a symptom. Reviewers had time to read properly again.
• ▹The regression rate dropped.
• ▹One thing got worse. In the first week the agents sat idle more than we'd expected. "Switch to useful work" only works if there's useful work ready to pick up. Review-cost-reducing tasks need their own backlog and their own priority. If they don't have one, the blocked branch ends in Idle and you've built an expensive waiting room.

There's one more thing I'd watch for. Idle agents look like waste on a cost dashboard, and sooner or later someone will want to raise the cap. Before anyone does, measure review time per PR and the regression rate. Those are the two numbers the cap is there to protect.

Where else this applies

• ▹429s and the planner. When the LLM API rate-limits you, does your planner keep fanning subtasks into a queue that can't drain? The same three parts apply one layer down.
• ▹Agents that can't say "too large". An agent that takes every task no matter the size is a consumer with no way to push back. A structured way for agents to refuse or split a task is backpressure at the task level.
• ▹Context-window saturation. The context window is a bounded queue. As it fills up, model quality tends to slip quietly well before you reach the hard limit, just like the reviewer did.

Design-review checklist

• ▹Who is your slowest consumer? Write a rough service rate next to every stage, people included.
• ▹Is the queue in front of that consumer bounded? Give the actual number and say where it's enforced. It should be in code, not convention.
• ▹What signal goes upstream? How does a producer learn it's blocked before it does the work?
• ▹What does a blocked producer do? It can wait, shed load, degrade, or switch to work that raises the consumer's service rate. Is that work lined up in advance?
• ▹Who sees it when the bound is hit? You need a metric, a dashboard and an owner. If the bound is hit all the time, your capacity is wrong. The bound probably isn't.

The reframe

If the consumer is fixed, adding producers doesn't raise output. You get a longer queue, longer waits, and a consumer under more pressure to cut corners. Before you spin up a twenty-first agent, find your slowest consumer and ask what would make it faster. Tomorrow's lesson picks up from here.

Extend your knowledge

• ▹Read Fred Hebert's essay "Queues Don't Fix Overload" on ferd.ca. It's short, and it explains clearly why bounding the queue is the real fix.
• ▹Read Eliyahu Goldratt's The Goal for the Theory of Constraints. Every time it says "factory floor", read "agent pipeline".
• ▹Look at the Reactive Streams specification and Elixir's GenStage. Both build demand-driven backpressure into the protocol itself.
• ▹Exercise: draw your own agent pipeline, put a service rate on every stage (people included), and run Little's Law on the queue in front of the slowest one.