Why Adding 12 More Subagents Made Our Pipeline Slower, Not Faster

Day 17: Your Orchestrator Agent Is a Hot Partition

Here's the part that stings: if every subagent's output has to route back through one planner before anything else can happen, bolting on more subagents doesn't buy you more throughput. It buys you a longer line in front of the one node that was already your ceiling.

The moment queue depth started climbing

Here's the pipeline: a planner agent breaks a ticket into subtasks, fans them out to N subagents, then merges the results and decides what happens next. Latency was mediocre, so the obvious move was going from 4 subagents to 16. Throughput should scale — that's the whole pitch for fanning out, right? Instead p95 latency got worse. Not flat. Worse. I sat there watching the orchestrator's inbound queue depth climb steadily while four times as many subagents sat idle — finished, waiting on a decision that hadn't landed yet. That's the moment it clicked: we hadn't parallelized the pipeline. We'd parallelized the one part that was never the bottleneck.

Callback to Day 16

Quick refresher, since this lesson leans on it: a hot row (or hot partition) is what happens when a sharded database still funnels a disproportionate share of writes through one key. Sharding only helps if the load actually spreads across shards — not just in principle, in practice.

The pipeline we actually had

Here's the shape of it. One planner agent decomposes the task and fans out to subagents running in parallel. Each subagent does real work on its own — reads code, writes a diff, runs a test. But nothing counts as 'done' until it passes back through the planner, which reasons over the result and decides pass, fail, retry, or merge, then picks what runs next. Every path through the graph, no matter how many subagents you stack on the fan-out side, converges back onto that one serial node on the fan-in side.

Same failure mode, new name: reasoning skew

A hot row is lock contention: many writers, one row, one lock, everything serialized. Our orchestrator had the same problem under a different name — call it reasoning skew. Many subagents, but one LLM call that has to read every result, hold the entire task state in context, and decide one thing at a time. The subagents parallelize fine because they don't share state with each other. The planner can't, because by design it's the one place all that state converges. You can 4x the subagents feeding it. You cannot 4x the planner's own reasoning — it's still one call, one context window, one decision at a time. Same as a hot row is still one lock no matter how many transactions are queued behind it.

How we diagnosed it like a DB engineer would

The instinct when latency goes bad is to count agents and check per-agent latency. Wrong axis entirely. We stopped counting agents and started watching the one metric that actually predicts throughput in a fan-in system:

• ▹Queue depth at the orchestrator's inbox, tracked over time — not the average, the trend. A climbing queue under steady load is the tell, exactly like a saturated hot partition.
• ▹Orchestrator wall-clock time per decision — how long that single LLM call takes to read merged context and output a decision, separate from subagent execution time.
• ▹Subagent idle time waiting on a decision — this is the number that made the '4x subagents' call look bad in hindsight. Most of those 16 agents were paid-for compute sitting idle.
• ▹Context length fed into the planner per turn — as more subagents report back, the merge context grows, which quietly slows the one call everything else depends on.

The naive fix vs. the real fix

The naive fix — more subagents — treats this as a capacity problem on the fan-out side. It isn't. The real fix is the same move you'd make on a hot row: shard the hot node's responsibility, or let callers resolve locally instead of always reporting back to the center.

• ▹Sharded the orchestrator's decision space — split 'merge and decide' into independent decision domains (test pass/fail is its own gate, code style is its own gate), so each becomes a smaller, cheaper, parallel LLM call instead of one giant serial one.
• ▹Let subagents self-resolve locally when the decision didn't actually need central context — a subagent that fails its own unit test doesn't need the planner's permission to retry. It retries, and only escalates after repeated failure.
• ▹Only routed to the planner what genuinely needed cross-subagent context — conflicting file edits, budget or priority tradeoffs. Everything else stayed local.
• ▹Cut the planner's per-turn context by having subagents report structured deltas instead of full transcripts, so the one serial call stayed fast even as N grew.

BEFORE (hot orchestrator):
  planner.decide(all_subagent_outputs) -> next_step   # one call, grows with N

AFTER (sharded decisions + local resolution):
  for subagent in subagents:
      if subagent.can_self_resolve(): subagent.retry_or_finalize()
      else: escalate(subagent.output)

  gate_tests.decide(test_outputs)        # parallel, narrow context
  gate_style.decide(style_outputs)       # parallel, narrow context
  planner.decide(only_cross_cutting_conflicts)  # small, rare, still serial — but no longer the whole pipeline

The generalizable rule (setting up Day 18)

Any node that every other node has to report back to before work can proceed — a planner agent, a human approver, a single 'merge' service — inherits this constraint, whether it's a database row, an LLM call, or a person in Slack. Spot it before you build it: ask whether that node's per-item cost grows with the number of parallel workers feeding it. If yes, you've designed a hot partition, full stop. And the fix is never 'add more workers upstream' — it's 'reduce what has to pass through the one node, or split its job into independent pieces.' That's the lens we'll turn on coordinator nodes generally — human or agent — in Day 18.

Extend your knowledge

• ▹Re-read Day 16's hot-row material with this lesson in mind — the same diagnosis steps (measure the concentrated node, not the distributed ones) apply directly.
• ▹Check how your current agent framework's orchestrator or supervisor node handles context growth as subagent count increases — that per-turn context size is your early warning signal.
• ▹Look at how load balancers avoid hot-node collapse in LLM inference serving (routing strategies that avoid piling requests onto one instance) — the same 'don't funnel everything through one queue' principle applies to agent coordination.
• ▹Before Day 18, map any 'approval' or 'review' step in your own pipeline — agent or human — and check whether its workload scales with upstream parallelism. That's next lesson's diagnostic.