Human-in-the-loop for multi-agent systems

Day 26: The Human Is an Agent Too (Designing Human-in-the-Loop for Multi-Agent Systems)

Your multi-agent pipeline is probably not failing. It's waiting. Most pipelines don't stall because an agent got something wrong. They stall because every agent can send a question to the same person, and that person is the slowest and least available part of the system. If you design human-in-the-loop (HITL) as a supervisor sitting above the loop, the pipeline ends up autonomous on paper and idle most of the time.

Here's today's idea in one sentence. Model the human as one more agent in the topology, with the highest latency, the scarcest bandwidth and the worst uptime of any agent. Then design the protocol around those limits, the same way you would for a slow, rate-limited external service.

1. The 37-escalation Monday

Here's how it plays out. Over the weekend the pipeline (planner, two coders, reviewer) works through a backlog. On Monday morning one engineer opens their laptop and finds 37 escalations. Nothing crashed, no agent errored, and every trace is green. The pipeline didn't break. It waited on 37 separate branches for a person who was asleep, then in standup, then in a 1:1.

All of it came from one line in the orchestrator system prompt:

If you are uncertain about any decision, ask the human before proceeding.

It looks like the responsible thing to write, but it has three problems. 'Uncertain' is never defined, and an LLM is always a bit uncertain. Asking costs the agent nothing. And 'before proceeding' puts the human on the hot path, so the run blocks until they reply. Each of the four patterns below fixes one of these problems.

2. The reframe: redraw the topology with the human as a node

Take the topology from the orchestration days. Stop drawing the human as a box above everything with arrows coming down from it. Draw them as a peer node with a spec sheet, the way you'd label an LLM endpoint with p95 latency and rate limits.

Now read that spec as if a vendor had sent it to you. Responses take hours. The service is offline two-thirds of the day. Throughput is a couple of hours of attention per day, and every call pays a large cold-start cost in the form of a context switch. You'd never put a service like that synchronously on the critical path of every request. You'd put a queue in front of it, batch the calls, cache what you can, set timeouts with fallbacks and keep a call budget. Those are the four patterns. HITL design is ordinary distributed-systems design applied to a component that happens to be a person.

• ▹High latency: never block on it synchronously. That's Pattern 2, the async queue.
• ▹Scarce bandwidth means you ration calls (Pattern 1, escalation budget) and make each one cheap to answer (Pattern 3, decision packet).
• ▹Low uptime. Every call needs a fallback for when nobody answers (Pattern 4, timeout defaults).
• ▹Context switches are expensive, so batch. The human should pay the switching cost once per session, not once per question.

3. Why agents push decisions upward

With a single agent, an occasional 'let me check with you' does no harm. Add more agents and things start to compound.

• ▹Uncertainty stacks. The coder isn't sure what the planner meant, and the reviewer isn't sure how the coder read it. Every handoff adds ambiguity, so the number of moments where an agent could reasonably feel unsure grows with depth and fan-out.
• ▹Escalating is the cheapest move available. Deciding means owning a mistake that a reviewer agent or an eval might catch, while escalating hands the risk to someone else for free. An LLM tuned to be careful and helpful will pick the zero-cost, safe-looking option whenever you let it.
• ▹Nobody owns the leftovers. With several agents and one human, each agent can reasonably assume someone else will resolve the ambiguity. The human is the only node every agent can reach, so they become the tie-breaker for everything.

A note from my PhD work: this is a known coordination failure, not a prompt bug, and the multi-agent systems literature on adjustable autonomy studied exactly this problem. Scerri, Pynadath and Tambe's work on the Electric Elves project (JAIR, 2002) showed that a one-shot 'ask the human or act' decision breaks down in practice. What worked was a transfer-of-control strategy: ask, wait a bounded time, then act or hand control to someone else. You can't fix this by rewording the prompt. You fix it by changing the protocol so that escalation has a price, an expiry and a default.

4. Pattern 1: Escalation budget

What it is: each task gets a fixed number of human interrupts, and agents spend them deliberately. Once the budget runs out, further decisions fall back to a written policy instead of going to the human.

Why it exists: it puts a price on asking. Once escalation costs something, the agent has to compare 'is this worth one of my two interrupts?' with 'can I decide this myself under the policy?'. That comparison is the reasoning you wanted from the start.

escalation_policy:
  budget:
    per_task: 2            # human interrupts a single task may spend
    per_run: 10            # hard cap across all tasks in one pipeline run
    shared_by: task        # planner, coders, reviewer draw from the same task budget
  must_escalate:           # these always go to the human and don't consume budget
    - irreversible_action  # prod data migration, external email, payment, deletion
    - security_boundary    # auth, secrets, permissions
  never_escalate:          # agents decide these, full stop
    - naming_and_style
    - test_structure
    - choice_between_equivalent_libraries_already_in_repo
  over_budget_fallback:
    reversible: proceed_with_recommendation
    irreversible: take_safe_option   # usually: don't do it, open a ticket
  log_every_decision: true           # including ones the agent made under fallback

Then replace the one-liner with an instruction the agent can actually follow:

You have {remaining_budget} human escalations left for this task.
Escalate only if (a) the action is irreversible or touches a security boundary,
or (b) you have two or more options with materially different outcomes AND the spec
does not decide between them. Otherwise, decide, record your reasoning in the
decision log, and continue. If the budget is 0, apply the fallback policy.

• ▹When to use it: any pipeline where more than one agent can reach the human.
• ▹When not to: early exploration, while you're still writing the spec and want to see every ambiguity. Do a few runs with an unlimited budget, read what comes up, add the answers to the spec, then switch the budget on.
• ▹Pitfall: budgeting per agent instead of per task. Four agents with two interrupts each gives you eight interrupts per task, which is the original problem in a tidier format.
• ▹Pitfall: letting 'must_escalate' grow until it covers everything. Review that list as suspiciously as you'd review IAM permissions.

5. Pattern 2: Async queue plus batching

What it is: the human is never on the hot path. When an agent escalates, it parks the blocked branch with its full state checkpointed and writes a decision packet to a queue. The orchestrator keeps running every branch that doesn't depend on the answer. The human clears the queue in scheduled batches, for example twice a day.

Why it exists: synchronous HITL turns human latency into pipeline latency, one-for-one. With async, that latency only delays the branches that depend on the answer. Batching turns 37 context switches into two.

The tooling already supports this. Durable-execution engines like Temporal let a workflow wait on a signal for days without holding resources. LangGraph's interrupt-plus-checkpointer pattern lets a graph pause at a node and resume later from saved state. Your agent state must be serialisable and resumable. If an escalation can only be answered while the process that raised it is still alive, your HITL is synchronous no matter what the UI looks like.

• ▹Before: one question blocks everything behind it, and the human gets pulled out of their work three separate times.
• ▹After: independent work keeps moving, only the parked branch waits, and the human spends one focused block on the queue.
• ▹Pitfall: everything turns out to be dependent. If your planner emits a strict linear chain, nothing runs while one step is parked. Go back to task decomposition and get the planner to emit a DAG with branches that are actually independent.
• ▹Pitfall: no urgent lane. Have one, require a justification for using it, and count how often it's used. If 'urgent' is more than a small fraction of escalations, it has become the default.

6. Pattern 3: The decision packet

What it is: the standard payload an agent must send when it escalates. It has to make sense on its own, so the human can decide in under a minute from the packet alone.

The rule: if the human has to open another tab (the PR, the trace, the ticket, Slack) to understand the question, the packet failed. Measure how often that happens and treat each case as a bug in the agent.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "DecisionPacket",
  "type": "object",
  "required": ["id", "task_id", "raised_by", "question", "options", "recommendation", "already_tried", "blast_radius", "reversible", "timeout", "default_action"],
  "properties": {
    "id": { "type": "string" },
    "task_id": { "type": "string" },
    "raised_by": { "type": "string", "description": "planner | coder-a | reviewer ..." },
    "question": { "type": "string", "maxLength": 280, "description": "One decision, phrased so it can be answered by picking an option" },
    "context": { "type": "string", "maxLength": 1200, "description": "Only what is needed to decide; inline the relevant snippet, not a link" },
    "options": {
      "type": "array", "minItems": 2, "maxItems": 4,
      "items": {
        "type": "object",
        "required": ["key", "description", "consequence"],
        "properties": {
          "key": { "type": "string" },
          "description": { "type": "string" },
          "consequence": { "type": "string" }
        }
      }
    },
    "recommendation": {
      "type": "object",
      "required": ["option_key", "confidence", "reason"],
      "properties": {
        "option_key": { "type": "string" },
        "confidence": { "type": "string", "enum": ["low", "medium", "high"] },
        "reason": { "type": "string" }
      }
    },
    "already_tried": { "type": "array", "items": { "type": "string" }, "description": "Spec sections read, tools called, alternatives ruled out" },
    "blast_radius": { "type": "string", "description": "What breaks, for whom, if this is wrong" },
    "reversible": { "type": "boolean" },
    "timeout": { "type": "string", "description": "ISO-8601 duration, e.g. PT8H" },
    "default_action": { "type": "string", "enum": ["proceed_with_recommendation", "take_safe_option", "abort_branch"] },
    "budget_remaining_after": { "type": "integer" }
  }
}

• ▹A recommendation with a confidence level is mandatory. It forces the agent to do the thinking, and it turns most reviews into 'approve recommendation', which takes seconds.
• ▹already_tried stops the human from suggesting something the agent has already ruled out. That round-trip is one of the most common ones you'll see.
• ▹blast_radius and reversible tell the human how closely to look. A reversible tweak to an internal test helper shouldn't get the same scrutiny as an irreversible email to 4,000 customers.
• ▹Validate every packet against the schema in the orchestrator. Send malformed packets back to the agent so nothing reaches the human with fields missing.
• ▹Pitfall: dumping the whole trace into 'context'. Compressing the decision is the agent's job. If it can't do that, the task is probably underspecified.

7. Pattern 4: Timeout defaults

What it is: every escalation says what happens if nobody answers before the timeout. This is the transfer-of-control idea from adjustable autonomy turned into a rule: ask, wait a bounded time, then act.

There are three defaults to choose from:

• ▹proceed_with_recommendation: for reversible actions where the agent's confidence is medium or high. No answer counts as approval. Most escalations should end up here.
• ▹take_safe_option covers reversible actions that carry more risk, and anything with low confidence. The agent takes the conservative branch (flag off, existing behaviour kept, optional refactor skipped) and logs it for later review.
• ▹abort_branch is for irreversible actions. The agent doesn't act. It closes the branch cleanly, opens a ticket and lets the rest of the run continue.

Separate reversible actions from irreversible ones, because irreversible actions are the only place where a hard block is worth what it costs. Deleting production data, sending external communication, moving money, changing permissions and running a schema migration with no rollback path are cases where waiting costs less than being wrong. Almost everything else an agent fleet does in a codebase can be undone with git, flags or a redeploy, so those actions should get a default that keeps work moving.

def resolve(packet, answer=None, now=None):
    if answer is not None:
        return answer
    if now < packet.raised_at + packet.timeout:
        return WAIT  # branch stays parked, others keep running
    if not packet.reversible:
        return ABORT_BRANCH  # the only hard stop
    if packet.default_action == "proceed_with_recommendation":
        return packet.recommendation.option_key
    return SAFE_OPTION

• ▹Pitfall: making every default 'abort'. You get the 37-escalation Monday again with extra steps. Instead of waiting, the work quietly dies.
• ▹Pitfall: letting the agent judge reversibility for itself. Classify actions in the tool layer, not in the prompt. A tool that sends email is irreversible whatever the agent thinks.

8. The metric nobody tracks: escalations per task and override rate

Teams instrument tokens, latency and pass rates for their agents, then leave the most expensive component, the human, unmeasured. Track two numbers every sprint:

• ▹Escalations per task = human-bound packets / completed tasks.
• ▹Override rate = decisions where the human chose something other than the agent's recommendation / decisions the human actually answered.
• ▹Also log the timeout rate (the human never answered) and the 'had to open another tab' rate from Pattern 3.

SELECT
  sprint,
  COUNT(*)::float / NULLIF(t.completed_tasks, 0)                    AS escalations_per_task,
  AVG(CASE WHEN p.human_choice IS NOT NULL
           THEN (p.human_choice <> p.recommended_option)::int END)  AS override_rate,
  AVG((p.human_choice IS NULL)::int)                                AS timeout_rate
FROM decision_packets p
JOIN (
  SELECT sprint, COUNT(*) AS completed_tasks
  FROM tasks
  WHERE status = 'completed'
  GROUP BY sprint
) t USING (sprint)
GROUP BY sprint, t.completed_tasks
ORDER BY sprint;

Here's how to read them:

• ▹High escalations, low override rate: agents are asking about things they already get right, and the human is just rubber-stamping. Tighten the budget and add to never_escalate.
• ▹High escalations and a high override rate mean the agents really are getting it wrong, usually because of the spec. Fix the spec or the agent's context, not the budget.
• ▹Low escalations, high override rate: this is healthy. The few questions that come up need a human.
• ▹Low on both? Check the timeout rate and spot-check the fallback decisions. Either things are fine, or agents are quietly making decisions they shouldn't.

When we instrumented this at PhoenixDX, the first picture was uncomfortable. Escalations per task were high and the override rate was low, so most of what reached a human was an agent asking permission to do what it was going to do anyway. We cut budgets, moved a batch of recurring question types into the spec and the never_escalate list, and made the packet schema mandatory. Escalations per task dropped sprint over sprint and the override rate went up. That's the direction you want, because what still reached a person was a real decision. I'm describing the shape and not quoting numbers, because the absolute values depend heavily on task mix. Watch the direction. If escalations per task aren't falling, your HITL setup has quietly become a human doing the whole loop.

9. Checklist: apply this to your pipeline this week

• ▹Budget: can you state the maximum number of times one task can interrupt a human? Is that limit enforced in code rather than in a prompt?
• ▹Queue: if the human goes offline for 16 hours, which branches keep running? If the answer is 'none', your human is on the hot path.
• ▹Packet: pull your last five escalations. Could each one be answered without opening another tab? Did each include a recommendation and a reversibility flag?
• ▹Default: does every escalation type have a written timeout action, with hard blocks reserved for irreversible actions classified at the tool layer?
• ▹Metric: do you know this sprint's escalations per task and override rate? Is escalations per task lower than last sprint, and does the override rate show that the escalations still coming in are real decisions?

10. Bridge to Day 27

Once the human is a scheduled, budgeted agent with a clean interface, every decision they make becomes structured data: a packet, an option, a recommendation and an override. That leads to tomorrow's question. How does the system learn from those decisions so the same escalation doesn't come back next week? Day 27 covers turning human decisions into spec updates, policy rules and few-shot memory, and doing it without overfitting to one reviewer's mood on a Monday.

Extend your knowledge

• ▹Read Scerri, Pynadath and Tambe, 'Towards Adjustable Autonomy for the Real World' (Journal of AI Research, 2002). This is where the transfer-of-control-with-timeouts idea comes from, out of the Electric Elves project, and it maps closely onto LLM agent fleets.
• ▹Eric Horvitz, 'Principles of Mixed-Initiative User Interfaces' (CHI 1999), covers when a system should act, ask or stay quiet, framed in terms of the cost of interrupting a person.
• ▹Build Pattern 2 on a durable runtime. Try LangGraph's human-in-the-loop interrupt with a checkpointer, or a Temporal workflow that waits on a signal, and check that a parked branch survives a process restart.
• ▹Rob Ewaschuk's 'My Philosophy on Alerting', written from his time as a Google SRE. Read 'escalation' wherever he writes 'page' and most of it applies directly to HITL design.