The orchestrator agent

Day 28: The Orchestrator Agent (and Why It Should Be Boring)

We gave our orchestrator the best model we had, and it lied to us. It wasn't malicious. It just marked a task "implemented and verified" when nothing had been verified. Once you go from one agent to several, something has to decide who does what and when the work is finished. How you design that coordinator matters more for reliability, cost and debuggability than which model you pick for the workers.

The moment we noticed

At PhoenixDX we ran a multi-agent setup for engineering tasks. One orchestrator split the work, and sub-agents did the coding, testing and reviewing. We put our strongest model on the orchestrator and wrote it a long prompt telling it to act like a brilliant architect. For a while it looked great.

Then someone read a trace end to end. A sub-agent had returned a patch with a failing test. The orchestrator didn't send it back. It started fixing the code inline, then pulled in more files so it could understand its own fix. Its context window filled up with details that belonged to the worker. Close to the limit, it summarised its half-finished edit as "implemented and verified" and marked the task done. Nothing had verified anything.

Our first instinct was to upgrade the orchestrator's model. That made it worse. The stronger model was better at doing the work itself, so it delegated less and absorbed more. The fix turned out to be making the orchestrator dumber.

Definition in plain words

Earlier in the series we covered agents (an LLM in a loop with tools) and sub-agents (agents spawned with their own fresh context to handle one piece of the work). An orchestrator agent is the coordinator above them. It decides who does what and when the work counts as finished. It doesn't do the work.

• ▹An air traffic controller doesn't fly planes. They sequence, route and clear them. If they grabbed the controls of one plane, every other plane in the sky would be without a controller.
• ▹A kitchen expediter doesn't cook. They call tickets, send dishes to stations and check every plate before it leaves the pass. A wrong plate goes back to the station, not into the expediter's own pan.

The three jobs an orchestrator has

• ▹1. Decompose: split the goal into tasks small enough to fit comfortably in a fresh sub-agent context. If a task only makes sense with the whole project history, you haven't split it enough.
• ▹2. Dispatch: send each task to the right worker with a clear brief (inputs, constraints, expected output). Track it. Retry, reassign or kill anything that stalls, loops or times out.
• ▹3. Accept: check each result against a definition of done before marking it complete. "Tests pass", "the schema validates" or "a reviewer agent signed off" are typical checks. "The worker says it's done" is not one.

Everything else belongs to the workers: writing code, researching, reasoning through a nasty bug, choosing an implementation. If the orchestrator is doing any of that, it's no longer an orchestrator.

Why a smart orchestrator backfires

Our trace showed three failure modes, and every one of them gets worse as the orchestrator gets more capable.

• ▹It absorbs work. A capable model that can see a problem will try to solve it, and delegating quietly turns into doing. You lose the worker's fresh, focused context, and the orchestrator becomes one slow, expensive worker that also coordinates on the side.
• ▹It hoards context. Every inline fix, every file it opens, every debugging detail lands in the orchestrator's window. That window is supposed to hold the plan and the status of every task. Fill it with one worker's details and the orchestrator loses track of the rest. When the context gets compacted or truncated, the plan state is often the first thing to go.
• ▹It rubber-stamps. Once the orchestrator has put its own reasoning into a result, it's a poor judge of that result. It reads "looks finished" as "is finished" because it partly wrote the thing. It's the same reason authors don't approve their own PRs.

The demotion: before and after

We didn't rewrite the system. We kept taking responsibilities away from the orchestrator until what was left could be replayed from a log.

BEFORE (smart orchestrator)          AFTER (boring orchestrator)
----------------------------         ---------------------------
prompt: "You are a brilliant         prompt: short, procedural.
  architect. Plan, implement,          "Pick next task. Route it.
  fix, and verify as needed."          Record result. Never edit."

planning: free-form, re-plans        task list: explicit, versioned,
  mid-flight in its own head           each task has a DoD

can edit worker output               read-only on deliverables

routing: "use judgment"              routing table: task type -> worker

stalls: hopes for the best           retry x2, timeout 10m, then kill
                                       and escalate to a human

acceptance: its own opinion          acceptance: tests / schema checks /
                                       reviewer agent in a FRESH context

log: chat transcript                 log: structured decisions you can
                                       replay (task, worker, verdict, why)

A lot of the "after" column doesn't need an LLM at all. Routing tables, retry counters and timeouts can be plain code. Our rule of thumb: deterministic code wherever you can, an LLM only where you must (turning a vague goal into a task list, say). Keep that LLM step small, and log it.

The bigger idea: boring center, smart edges

This isn't a new idea. Keep the center boring and put the intelligence at the edges.

• ▹Kubernetes: the control plane schedules work and reconciles desired state against actual state. It doesn't run your workload. Pods do.
• ▹Message brokers and microservices: James Lewis and Martin Fowler's 2014 microservices article called it "smart endpoints and dumb pipes". The broker routes. The services think.
• ▹Multi-agent systems research has been arguing about thin versus thick coordinators for decades. The Contract Net Protocol (Reid G. Smith, 1980) is the classic thin version. A manager announces tasks, contractors bid on them and do the work, and the manager mostly awards and accepts. LLM agents bring the same trade-offs back with a new kind of node.

An honest caveat: when a thicker orchestrator is worth it

For some tasks, the decomposition is the hard part. Open-ended research is the obvious case: you don't know what the subtasks are until early results come in. Anthropic's write-up of their multi-agent research system describes a lead agent that plans and adapts as subagents report back. For that kind of work, it's a reasonable design.

Even then, split the roles. One agent is the planner: it thinks hard and produces or updates the task list. A separate component is the dispatcher: it routes, enforces timeouts, retries and runs acceptance. The planner gets to be smart and the dispatcher stays boring. You get intelligent decomposition and still keep a control loop you can audit. And the planner still never touches deliverables.

When NOT to use an orchestrator at all

• ▹If the task fits in one agent's context and finishes in a few minutes, use one agent with good tools. It's simpler, cheaper and easier to debug.
• ▹If the steps are fixed and known up front, a plain pipeline or workflow script will coordinate better than any LLM.
• ▹Use an orchestrator when the work genuinely fans out into independent tasks that benefit from separate contexts.

Five questions to ask about your orchestrator

• ▹Does it ever write the deliverable itself? Then it's absorbing work. Make it read-only on outputs.
• ▹Can you replay its decisions from logs? If you can't reconstruct why task X went to worker Y and got accepted, it isn't auditable.
• ▹Does every task it hands out come with a definition of done? Without one, acceptance is guesswork.
• ▹Is acceptance checked by something other than the agent that did the work? Tests, validators or a reviewer in a fresh context. Never self-approval.
• ▹What happens when a sub-agent hangs? If the answer is "it waits", you need timeouts, a retry limit and a kill-and-escalate path.

Next up

Once the orchestrator is thin, it can't paper over weak handoffs anymore. System quality then comes down to what passes between agents: the brief that goes out and the result that comes back. Tomorrow, Day 29, covers delegation briefs and handoff contracts. We'll look at how to write a task brief a fresh sub-agent can act on, and how to define a result format the orchestrator can accept or reject without having to think.

Extend your knowledge

• ▹Read Anthropic's "Building effective agents" post. Focus on the orchestrator-workers pattern and the advice to start with the simplest workflow that works.
• ▹Read Anthropic's engineering write-up on how they built their multi-agent research system. It shows when a planning-heavy lead agent pays off, and what it costs you.
• ▹Look up the Contract Net Protocol (Reid G. Smith, 1980). It's the thin-coordinator idea from classic multi-agent systems research, long before LLMs.
• ▹Exercise: take one multi-agent trace from your own setup and mark every step where the orchestrator did something other than decompose, dispatch or accept. Each mark is a candidate to move into a worker or into deterministic code.