Evaluating agent handoffs

Day 29: Evaluate the Edges, Not Just the Exit

Two runs of the same pipeline both scored 0.92. One shipped without trouble. The other broke three internal callers the next morning, and our eval had no way to tell them apart. An end-to-end score tells you that something broke. It can't tell you where. In the multi-agent systems I've worked on, most failures that reached production started at a handoff between two agents. This lesson covers how to evaluate those handoffs one at a time.

Two runs, both 0.92

Here's the full case. Same planner → coder → reviewer pipeline, similar tickets. Both runs scored 0.92 on our end-to-end eval: tests green, reviewer approved, diff looked clean. One went out quietly. The other is the one that broke the callers.

The final outputs didn't explain the difference. So we opened the trace for the bad run and read every message between agents, line by line. The bug was in the second message:

# INPUT TO PLANNER (the ticket)
Add retry with exponential backoff to the payment webhook handler.
You MUST NOT change the signature of handle_webhook(event, ctx).
Three internal services call it positionally.

# EDGE: planner -> coder (step 2 of 4)
Implement retry with exponential backoff in handle_webhook.
Prefer keeping the handle_webhook signature stable where possible.
The retry policy should be configurable per call.

"MUST NOT" had turned into "prefer". The planner also slipped in an idea of its own, "configurable per call", which pushed toward a signature change. The coder did the reasonable thing: it added a retry_policy parameter in front of ctx and wrote a note explaining the trade-off. The reviewer saw "prefer… where possible" next to a sensible justification and approved. Every agent behaved well given what it was told. You'd only see the problem by putting the ticket and the handoff side by side and reading them word by word.

Why the score couldn't see it

An end-to-end eval grades the last agent's output, and a strong downstream agent can build a plausible result out of a bad input. Each hop adds polish, structure and confident wording. None of that adds accuracy. By the time the reviewer approves, a softened constraint reads like a deliberate design decision. In the handoff design lesson we treated the message between agents as the product. This is the testing side of that idea: if the message is the product, you have to inspect the message itself, not just whatever comes out at the end.

The reframe: every edge is an API

Draw your agent graph. Every arrow is a call from a producer to a consumer, with a payload and an implied contract. Nobody would ship a set of microservices with one end-to-end test and no contract tests. Yet that's how many of the multi-agent evals I've seen are set up: one probe at the exit, nothing on the edges.

An edge contract asks three questions about a single message, and each one becomes a check:

• ▹Dropped: did every hard constraint from upstream make it into this message, at the same strength?
• ▹Invented: is the message claiming anything with no source upstream or in a tool output?
• ▹Sufficient: could the receiving agent do its job from this message alone?

Edge check 1: Dropped

List the hard constraints in the upstream input. Then check that each one shows up in the outgoing message at the same strength. The check is cheap and mostly deterministic, and it would have caught the incident above. Hard constraints tend to use a small set of modal words and usually point at a specific identifier, so you have something concrete to match on.

import re

HARD = re.compile(r'\b(must not|must|never|do not|only|always|required)\b', re.I)
SENT = re.compile(r'(?<=[.!?])\s+')
IDENT = re.compile(r'\b[a-z]+_[a-z_]+\b|\b\w+\(')

def constraints(text):
    return [s for s in SENT.split(text) if HARD.search(s)]

def dropped_check(upstream, handoff):
    findings = []
    for c in constraints(upstream):
        keys = {k.rstrip('(') for k in IDENT.findall(c)}
        if not keys:
            findings.append(('NEEDS_LLM', c))  # no anchor, grade semantically
            continue
        hits = [s for s in SENT.split(handoff) if any(k in s for k in keys)]
        if not hits:
            findings.append(('DROPPED', c))
        elif not any(HARD.search(h) for h in hits):
            findings.append(('SOFTENED', c, hits))
    return findings

# On the incident trace:
# ('SOFTENED', 'You MUST NOT change the signature of handle_webhook(event, ctx).',
#  ['Implement retry with exponential backoff in handle_webhook.',
#   'Prefer keeping the handle_webhook signature stable where possible.'])

Common pitfall: only checking that the constraint is present. In the incident, it was present. It had just lost its teeth. Strength is part of the contract, so SOFTENED is a failure, not a warning. If your planner emits structured output, give constraints their own field and compare that field directly. The regex is the fallback for free-text edges.

Edge check 2: Invented

The mirror-image failure is a handoff that adds facts nobody supplied: a confident file path that doesn't exist, a requirement the planner assumed, a test result from a run that never happened. In one way invented claims are worse than dropped ones. Downstream agents take them as ground truth and stop checking.

Follow that claim through. The coder never ran the test suite. "All tests pass" was a pattern it had learned, not something it observed. The reviewer repeated it. By the end the claim belonged to the reviewer, which made it sound more trustworthy, not less. One unsupported sentence crossed two edges and came out as a sign-off.

def invented_check(handoff, evidence, tool_calls):
    # evidence: upstream messages + tool outputs visible to this agent
    out = []
    if re.search(r'tests? (pass|passed|are green)', handoff, re.I):
        if not any(c.name == 'run_tests' and c.exit_code == 0 for c in tool_calls):
            out.append(('INVENTED', 'test result with no passing run_tests call'))
    for path in re.findall(r'[\w./-]+\.(?:py|ts|go|sql|ya?ml)', handoff):
        if path not in evidence:
            out.append(('INVENTED', f'path {path} not seen upstream or in tool output'))
    return out  # assumed requirements are fuzzy: send those to an LLM grader

Sort invented claims into two buckets. Checkable claims (test results, paths, numbers, API names) can be grounded against the trace with plain deterministic code. Fuzzy ones ("the user wants this configurable") need an LLM grader. Give it the upstream messages and the handoff and ask one question: for each claim in the handoff, quote its source, or say NONE.

Edge check 3: Sufficiency

Teams are usually surprised by this one. Give the receiving agent the message and nothing else: no shared scratchpad, no parent conversation, no cached repo summary. Can it still do the task? Plenty of pipelines only work because of leaked context. The coder can read the original ticket from shared memory, so a weak handoff never gets exposed. Then someone refactors memory, adds a parallel worker with a clean context, or swaps in a different model, and quality slides without anyone noticing.

# pseudocode: adapt to your framework's agent/session API
def sufficiency_check(receiver, handoff, task_check):
    isolated = receiver.fresh_session(memory=None, shared_context=None)
    result = isolated.run(handoff)
    return {
        'passes': task_check(result),            # same acceptance check as prod
        'questions': isolated.clarifying_questions(),  # what it had to guess
    }

• ▹Passes in isolation and in the full pipeline: the handoff stands on its own.
• ▹Fails in isolation but passes in the pipeline? You're relying on leaked context. Fix the message, not the agent.
• ▹Fails both ways: the handoff is broken, and your end-to-end eval should have caught it too. Find out why it didn't.
• ▹Don't throw away the clarifying questions. Each one points to a field your handoff schema is missing.

Fault injection on a single message

The three checks tell you whether your edges are clean today. Fault injection tells you whether the system would notice if one wasn't. Replay a real trace, corrupt exactly one message on one edge, let the rest of the run play out, and record who catches it, if anyone does.

MUTATIONS = {
    'soften': lambda m: re.sub(r'\bmust not\b', 'should try to avoid', m, flags=re.I),
    'drop':   lambda m: drop_field(m, 'acceptance_criteria'),
    'invent': lambda m: m + '\nNote: existing tests already cover retry behaviour.',
}

def inject(trace, edge, kind):
    run = trace.replay_until(edge)
    run.messages[edge] = MUTATIONS[kind](run.messages[edge])
    final = run.continue_from(edge)
    return {
        'edge': edge, 'mutation': kind,
        'e2e_score': e2e_eval(final),
        'caught_by': [a.name for a in final.agents if a.flagged(kind)],
    }

Run every mutation on every edge and lay the results out as a grid: edges down one side, mutations across the top, and in each cell the agent or eval that caught it, or "nobody". The first time we ran this, most cells said nobody, and the end-to-end score barely moved. That's the lesson. The "nobody" cells are your real risk map: each one is a place where a single bad message can ride all the way to production with nothing in its way.

How we wired it in at PhoenixDX

• ▹Start from the traces you already have. We didn't build a new harness. Our pipelines already logged every inter-agent message, so edge assertions were just functions over stored spans. The first ones came straight from real incident traces: each incident became one assertion on the edge where it started.
• ▹Each handoff got a fidelity score, shown on the dashboard right next to the end-to-end score. The formula is simple on purpose: start at 1.0, subtract for each dropped or softened hard constraint and each ungrounded checkable claim, and keep sufficiency as a separate pass/fail bit. When it goes red, an engineer opens the actual message.
• ▹We alert on divergence, not only on low scores. The signal worth paging on is edge fidelity dropping while the end-to-end score holds flat. That means downstream agents are covering for upstream mistakes, and that cover won't last.
• ▹The planner's outgoing edges came first. It sits at the root of the fan-out, so a mistake there reaches every worker and the reviewer. A coder→reviewer error hurts one branch. A planner error hurts all of them.
• ▹Fault injection is now part of how we change pipelines. Touch a prompt, model or memory setup on a watched edge, and the mutation grid reruns on a fixed set of traces before the change merges.

def edge_fidelity(dropped, softened, invented, n_constraints, n_claims):
    c = (len(dropped) + 0.5 * len(softened)) / max(n_constraints, 1)
    i = len(invented) / max(n_claims, 1)
    return round(max(0.0, 1.0 - c - i), 2)
# weights are a judgment call; keep them boring and stable so trends mean something

Honest limits

• ▹Edge evals cost tokens. The dropped check and checkable-claim grounding are nearly free. Sufficiency reruns an agent, and fault injection reruns most of the pipeline, so run those two on samples and on changes, not on every production run.
• ▹An LLM grader for invented claims has blind spots of its own. It tends to accept a claim that sounds like something upstream said, and its mistakes can overlap with those of the agents it's grading, especially within the same model family. Grading the grader is a separate problem, and this lesson doesn't solve it.
• ▹Regex extraction misses constraints written without hard modals, like "the API is frozen". Use structured planner output where you can and send anchorless constraints to semantic grading.
• ▹You don't need a check on every edge. My rule of thumb: start with edges that fan out (one producer, many consumers) and edges that cross a model or team boundary. In my experience wording drifts most where producer and consumer were built or tuned by different people, or run on different models.

Close: put a probe on every edge that matters

An end-to-end score is a smoke alarm. It isn't a map. Put a probe on every edge that matters and check three things: what got dropped, what got invented, and whether the receiver could have worked from the message alone. Then corrupt one message on purpose and see whether anything notices. Tomorrow, on the final day, we use all of this to answer the question the series has been building toward: once you can measure every handoff, does your multi-agent system actually beat one strong agent on the same budget?

Extend your knowledge

• ▹This week: take your planner's outgoing edge, pull a batch of recent traces, and run the Dropped check from this lesson over them. Read every SOFTENED hit yourself before you trust the regex.
• ▹Read 'Why Do Multi-Agent LLM Systems Fail?' (Cemri et al., 2025), the paper behind the MAST failure taxonomy. Its inter-agent misalignment category lines up directly with edge evals.
• ▹Spend some time on consumer-driven contract testing: the Pact framework docs and Martin Fowler's bliki entry on ContractTest. Then ask yourself what a 'consumer-driven' handoff schema would look like for your reviewer agent.
• ▹Anthropic's engineering post 'How we built our multi-agent research system' is worth a close read, especially the parts on how the lead agent writes task descriptions for subagents. Put those next to your own planner's handoffs and compare.