The Trace Was 100% Green — So Why Was the Answer Wrong?

Why this matters

A trace can be all green — every tool call fired, every latency inside spec — and the customer still walks away with a wrong answer nobody on your team can explain. That's the gap between logging what an agent did and recording why it did it.

The incident: a clean trace, a wrong answer

A support agent at PhoenixDX pulled a customer's order status, hit the right API, got a valid response back, formatted it cleanly. The tool call was correct. The answer was wrong — the customer was actually asking about a refund that had already superseded that order, and the agent reached for the wrong tool. Nothing in the trace was broken. We opened the dashboard expecting to find the bug in five minutes. We didn't.

Every span green. Every tool call succeeded. And we still couldn't tell the customer — or ourselves — why the agent made the choice it made.

What the trace had vs. what the postmortem needed

Standard agent observability tooling — LangSmith, Langfuse, most OTel-based setups — captures roughly the same shape of data out of the box: timestamp, tool name, input args, output, latency, maybe a token count. That's a record of what ran, full stop.

• ▹What we had: `get_order_status` called at 14:32:07, args `{order_id: 8842}`, 340ms latency, 200 response.
• ▹What we needed: at the moment before that call, the agent also considered `get_refund_status` and `escalate_to_human` — which one scored highest, and why did refund lose?
• ▹The trace could replay the action. It couldn't argue for or against the decision.

Name the pattern: action logs vs. reasoning records

Most 'agent observability' stops at the action layer because that's the cheap part to instrument — it's middleware wrapped around the tool-call interface. Intercept the function call and you get name, args, result, latency for free. It fills a dashboard, so it looks like observability. But it's a log of what the agent did, not a record of why it did that instead of something else.

This bites harder with agents than with traditional services, because the failure mode is different. A traditional service throws, or returns a wrong value from a deterministic bug — you bisect the code and find it. An agent's 'bug' is usually a reasonable-looking decision that turned out wrong given context the trace never captured. There's no stack trace for 'chose plausible option B over correct option A.'

The fix: instrument the decision point, not just the winner

If the tool call is the output of a decision, move your instrumentation one layer up — to the point where the model, or your routing logic, scores candidates and picks a winner. Capture what it considered, what it rejected, and whatever signal drove the ranking. Most of the time this is one structured log line, not a new pipeline.

{
  "decision_id": "dec_9f21",
  "trace_id": "tr_8842",
  "timestamp": "2026-09-20T14:32:06.9Z",
  "context_summary": "customer msg: 'my order 8842 refund never came'",
  "candidates": [
    {"tool": "get_order_status", "score": 0.71, "why": "order_id present in msg"},
    {"tool": "get_refund_status", "score": 0.68, "why": "keyword 'refund' matched, but no refund_id extracted"},
    {"tool": "escalate_to_human", "score": 0.22, "why": "low confidence fallback"}
  ],
  "chosen": "get_order_status",
  "why_not_others": "refund_id missing from entities, order_id was present and higher-confidence extraction"
}

That last field — `why_not_others` — is the whole point. It's what turns a postmortem from 'the tool call was technically valid' into 'the entity extractor failed to pull refund_id, which starved the refund tool of the signal it needed to win.' That's a root cause you can actually fix — tighten refund_id extraction, or add a tie-breaker rule for when order and refund keywords both show up.

The tradeoff, stated honestly

This isn't free. Logging candidates and rejection reasons on every decision means more storage, noisier traces, and in some setups an extra LLM call just to get the model to articulate why it passed on an option — models don't reliably surface this from a single forward pass. You may need to explicitly prompt for a ranked list with justifications, or pull raw logits/scores off your routing layer if an LLM isn't doing the routing at all.

• ▹You won't want this on every tool call in a high-volume agent — the storage and latency cost doesn't pay for itself on decisions that are almost never wrong.
• ▹You do want it on decisions with a high branching factor (3+ plausible tools), high customer impact (refunds, cancellations, anything customer-facing), or decisions your team has already been burned by once.
• ▹This is a budgeting problem, not an instrument-everything problem — tomorrow's lesson is specifically about deciding what earns this level of instrumentation and what doesn't.

The reframe

Next time you wire up tracing for an agent, ask one question before you ship it: does this let me argue with the agent's reasoning, or does it just let me replay its actions? If the honest answer is 'replay,' what you've built is an expensive video recorder, not observability.

Extend your knowledge

• ▹If you're on LangSmith or Langfuse, check whether your framework exposes a pre-tool-call hook or 'agent scratchpad' — that's usually the cleanest place to intercept candidate scoring before the winning tool call fires.
• ▹Read up on how your LLM provider's structured output / tool-choice APIs expose ranked or scored candidates — some routing setups return top-k tool suggestions with scores instead of a single choice, which is often the cheapest source of 'why not others' data you'll find.
• ▹Look at how your team runs postmortems on agent incidents today: if every one ends with 'the trace looked fine, we're not sure why it picked that,' that's the tell you're logging actions, not decisions.
• ▹Tomorrow's Day 15 lesson: how to prioritize which decisions earn this level of instrumentation, so you're not trying to log everything.