All Green, Totally Broken: What OTel Traces Can't See in a Multi-Agent Swarm

Day 19: Observability — Logs, Metrics, Traces, Rewritten for Multi-Agent Systems

Your dashboard can be lit up green while a swarm of your own agents quietly wrecks the thing they were supposed to produce. That's not a hypothetical — it's baked into what logs, metrics, and traces were built for: answering "was this request slow or broken," never "which agent decided what, in what order, and why." Bolt OTel onto a multi-agent pipeline and call it observability, and this is the exact gap that bites you first.

The incident

Six agents in one pipeline — a planner, three executors running in parallel, a merger, a verifier — spent about 20 minutes overwriting each other's shared state. Every service stayed up. p99 was fine. Error rate sat at zero, because nothing actually threw an error — each agent finished its own step successfully. Not one alert fired. A human caught it only because the final output was visibly wrong: half-finished edits stitched together with numbers that contradicted each other. We pulled up the dashboard. Green, top to bottom. Then we went looking for what had actually happened, and the standard toolkit had nothing to say.

The postmortem question that broke the toolkit

The question we needed answered was never "which span was slow." It was "which agent decided what, in what order, and why did agent C still believe a piece of state that agent A had already overwritten." OTel traces are built around request/response: one parent span, some children, timestamps, a single causal line from entry to exit. They'll tell you a call took 340ms and what it was waiting on downstream. They will not tell you that agent B acted on a belief about the world that was already stale, because two other agents had rewritten shared state in the meantime. Timing was never the problem here. A decision made on bad information, then carried forward through four more agents, was the problem — and nothing in the trace even recorded that a decision had been made.

Logs, metrics, traces — the one-line versions, and the gap

• ▹Logs: a timestamped record of one discrete event — what happened, here, right now.
• ▹Metrics: numbers aggregated over time — counts, rates, latencies. How much, how often, how fast.
• ▹Traces: a causal chain of spans showing how a single request moved through the system — what called what, and how long each step took.
• ▹All three assume the same thing: one straight causal line from a request coming in to a response going out.
• ▹Agent swarms don't work that way. Work branches across parallel agents, retries fire mid-flight, and handoffs are emergent — agent A often doesn't know it's about to hand off to agent C instead of B until a routing decision fires. A trace can show you the spans. It can't show you what each agent believed when it made its move.

The reframe: a flight recorder, not a faster dashboard

The fix isn't more spans or a shinier OTel exporter. It's a different artifact entirely: a black-box flight recorder for the swarm. A flight recorder isn't there to tell you the plane is flying smoothly — it exists so that after a crash, you can reconstruct exactly what every instrument showed and what the pilot decided, in order. Applied to agents, that means recording, at every point where control or state passes hands: what state that agent believed was true, what it decided to do, and why. Keep the dashboard — timing and errors are still useful. But they're not the instrument that answers the postmortem question. The decision record is.

What we actually added at PhoenixDX

After this incident, we started logging decisions at every handoff boundary — the moment control or a result moves from one agent to the next. Each entry holds three things: the state the handing-off agent believed was current, the decision it made, and a short reason trace — usually the relevant slice of its own reasoning or tool output — for why. Cheap to add, because handoff points are already a small, countable set in any agent pipeline; you're not instrumenting every LLM call, just the seams where responsibility changes hands. It caught the next version of this exact bug before it shipped: in staging, a decision log showed agent D acting on a document version that agent B's own log said had already been superseded two handoffs earlier. Same failure mode. Caught in minutes instead of surfacing as garbage output 20 minutes in.

The takeaway rule for tomorrow's build

If you can't replay "agent B's exact view of the world at turn 9" from your telemetry, you don't have observability for this system — you have observability for a different system that this one only resembles. Logs, metrics, and traces still earn their keep; keep all three. But for a multi-agent system, they're necessary and nowhere near sufficient. The unit of failure is a decision propagating through agents, not a slow span, so your telemetry has to capture causality between agents — who knew what, when, and what they did about it — not just timing inside one.

Extend your knowledge

• ▹Read OpenTelemetry's semantic conventions for generative AI systems to see how far OTel is stretching spans to cover LLM/agent-specific attributes — and where it still falls short of cross-agent causality.
• ▹Pick one real handoff boundary in a multi-agent system you own and add a decision log to it this week: believed-state, decision, reason. See what it catches in your next staging run.
• ▹Look at how agent frameworks with built-in tracing — LangSmith, or your own agent SDK's tracing hooks — represent multi-agent handoffs today, and check whether they capture belief state or just timing.
• ▹Pull up your last multi-agent incident postmortem and ask the Day 19 question retroactively: could you have answered 'which agent decided what, in what order, and why' from what you actually logged?