Observability for LLM apps

Day 28: Observability for LLM Apps — Log Decisions, Not Calls

Your agent will make a bad call in production while every dashboard you own stays green. That isn't a monitoring gap you can close with more alerts. Most LLM failures never throw. They show up as confident, plausible, wrong choices. If your telemetry only records which calls ran, you get a complete archive that still can't tell you why the system did what it did.

The all-green trace

Here's the kind of thing we hit at PhoenixDX. We had an internal ops agent for vendor invoices. It checked for duplicates, then posted or archived each one. One morning a finance lead asked why a real invoice had been archived as a duplicate. We opened the trace and it was green all the way down. Latency was normal. Every tool call returned 200. Nothing threw, and the token counts looked ordinary. Each LLM span carried its full prompt, about 40KB of system prompt, tool schemas, retrieved policy chunks and earlier tool outputs. Technically, the data was complete. In practice nobody could read it. Two engineers lost most of an afternoon scrolling through dumps. The trace could tell us what was called, in what order and how fast. We needed to know why it picked that, and the trace had no answer.

The mismatch: APM assumes failures throw

Traditional observability (APM, distributed tracing, error rates) assumes a failure leaves a mark: an exception, a 5xx, a timeout, a latency spike. LLM failures usually leave nothing. The model picks the wrong tool, trusts the wrong chunk or decides it's finished too early, and every layer underneath reports success. The real cause often sits a few steps before the visible mistake, too. Maybe a tool result looked like evidence, a retrieved chunk was off-topic, or the model dropped an argument. Spans show you what was called, not why. In an agent, the why is the bug.

The shift: make the decision your unit of telemetry

Stop recording 'one LLM call' as your unit. Record 'one decision' instead. You already know a version of this from the Architecture Decision Record (ADR), popularised by Michael Nygard. An ADR captures the context, the decision and its consequences so someone can later work out why. Many later ADR templates also list the options that were considered. Do the same thing automatically, every time the model chooses something:

• ▹Tool selection: which tool it called, and with which arguments.
• ▹Routing: which sub-agent, workflow or model gets the work.
• ▹Planning: the steps it committed to.
• ▹Final answer: what it told the user or wrote to a system, and whether it stopped or kept going.

One LLM call can hold one decision or several, and one decision can take a few calls. Calls are how the work happens. Decisions are what you debug.

The decision record schema

Most of what follows builds on this schema. Write one record per choice point:

{
  "decision_id": "dec_7f3a",
  "parent_decision_id": "dec_7f39",
  "run_id": "run_2026_10_13_0412",
  "step": 3,
  "decision_type": "tool_call",
  "options_available": ["search_invoices", "get_vendor", "mark_duplicate", "post_invoice", "escalate_to_human"],
  "choice": {
    "name": "mark_duplicate",
    "args": {"invoice_id": "INV-20931", "duplicate_of": "INV-18877"}
  },
  "stated_intent": "INV-18877 matches amount and date, so this is a duplicate; archive it.",
  "context_refs": [
    "chunk:policy_dupes@sha256:91c0...",
    "tool_out:search_invoices@sha256:4be2..."
  ],
  "model": "claude-sonnet-5-5",
  "prompt_version": "invoice-agent@v14",
  "outcome": {"status": "executed", "side_effect": "invoice_archived"},
  "human_signal": null,
  "raw_prompt_ref": "s3://llm-cold/run_2026_10_13_0412/step3.json",
  "ts": "2026-10-13T04:12:09Z"
}

• ▹decision_id: a stable ID for this choice, so other records can point at it.
• ▹parent_decision_id: the earlier decision whose output became this one's context. This link turns a pile of records into a tree you can walk.
• ▹step: where this sits in the run. Sort by it, and loops become obvious.
• ▹options_available: the tools, routes or candidates the model could actually see. You can't judge a choice without knowing what it was picked from.
• ▹choice: what it picked, arguments included. Many quiet bugs live in the arguments.
• ▹stated_intent: one short sentence from the model about what it's doing and what it expects. It's a claim, not an explanation.
• ▹context_refs: hashes or IDs of the chunks and tool outputs in the window. Store refs, not raw text. They're cheap to keep and easy to join and diff.
• ▹model + prompt_version: which model and which instructions made the call, so you can split results by deploy.
• ▹outcome: what actually happened next, such as executed, errored, a side effect or the final status.
• ▹human_signal: filled in later, when a person retries, edits, rejects or approves the result. It's null at write time.
• ▹raw_prompt_ref: a pointer to the full prompt in cold storage, for the rare times you really need it.

How to capture it cheaply

You don't need a new platform. You need three changes, and most of them live in your tool router:

• ▹Ask for intent next to the action. Add two short required fields, intent and reason, to every tool's input schema. The router strips them before running the tool and logs them on the decision record. Each is one sentence, so it costs a few dozen tokens per call.
• ▹Log context by reference. Hash every retrieved chunk and tool output, store the content once in a content-addressed store keyed by hash, and put only the hash in context_refs. If the same policy chunk shows up in 10,000 runs, you store it once and link it 10,000 times. Tag chunks in the prompt (e.g. [ref:4be2]) so the model can cite them in reason.
• ▹Demote the raw prompt. Keep it in cheap cold storage behind raw_prompt_ref, with short retention. Open it for one record when you need to. It's not where you start looking for a problem.

import hashlib, json, uuid

DECISION_FIELDS = {
    "intent": {"type": "string",
               "description": "One sentence: what you are about to do and what you expect to learn or change."},
    "reason": {"type": "string",
               "description": "Which context made you choose this. Cite ref ids like [ref:4be2]."},
}

def with_decision_fields(tool_schema: dict) -> dict:
    s = json.loads(json.dumps(tool_schema))
    s["input_schema"]["properties"].update(DECISION_FIELDS)
    s["input_schema"].setdefault("required", []).extend(["intent", "reason"])
    return s

def put_context(content: str, kind: str) -> str:
    h = hashlib.sha256(content.encode()).hexdigest()[:16]
    context_store.put_if_absent(h, content)          # stored once
    return f"{kind}@sha256:{h}"                        # linked many times

def route_tool_call(run, call, window_refs, parent_id):
    args = dict(call.input)
    intent, reason = args.pop("intent", None), args.pop("reason", None)
    decision_id = f"dec_{uuid.uuid4().hex[:8]}"

    result = TOOLS[call.name](**args)                 # real tool never sees intent/reason
    out_ref = put_context(json.dumps(result), f"tool_out:{call.name}")

    decisions.insert({
        "decision_id": decision_id,
        "parent_decision_id": parent_id,
        "run_id": run.id, "step": run.next_step(),
        "decision_type": "tool_call",
        "options_available": [t["name"] for t in run.visible_tools],
        "choice": {"name": call.name, "args": args},
        "stated_intent": intent, "reason": reason,
        "context_refs": list(window_refs),
        "model": run.model, "prompt_version": run.prompt_version,
        "outcome": {"status": "executed", "output_ref": out_ref},
        "human_signal": None,
        "raw_prompt_ref": cold_store.put(run.id, run.current_prompt()),
    })
    window_refs.add(out_ref)                          # next decision sees this output
    return result, decision_id

Before and after: replaying the incident

Before: you have four LLM spans, each carrying a 40KB prompt. You open the last one and find mark_duplicate. Then you scroll up through tool schemas and policy text, hunting for whatever looked like evidence. Then you repeat that for the span before it, and the one before that.

After: you start at the bad outcome and walk parent_decision_id backwards.

-- Walk back from the bad decision to the root of the run
WITH RECURSIVE chain AS (
  SELECT * FROM decisions WHERE decision_id = 'dec_7f3b'   -- archive_invoice
  UNION ALL
  SELECT d.* FROM decisions d
  JOIN chain c ON d.decision_id = c.parent_decision_id
)
SELECT c.step,
       c.choice->>'name'  AS tool,
       c.choice->'args'   AS args,
       c.stated_intent,
       r.ref               AS context_ref,
       LEFT(cs.content, 200) AS context_preview
FROM chain c
CROSS JOIN LATERAL jsonb_array_elements_text(c.context_refs) AS r(ref)
LEFT JOIN context_store cs ON cs.ref = r.ref
WHERE r.ref LIKE 'tool_out:%'
ORDER BY c.step;

The output takes about a minute to read. At step 3 the model marked the invoice as a duplicate. Its context had one new tool output: a search hit for INV-18877, from a different vendor but with the same amount and date. Step back to step 2. The stated intent says 'find invoices from the same vendor', yet the arguments contain no vendor_id. The bug isn't in the final action. It's in the gap between what the model said at step 2 and what it did, and a search tool that cheerfully returned fuzzy matches made it worse. The fix had two parts. We made vendor_id required in the tool schema, and we added a policy line saying a match only counts as a duplicate if the vendors match. Neither problem would ever have shown up on an error dashboard.

Three signals you get almost for free

• ▹Intent vs action drift. Compare stated_intent with the actual choice and arguments. Use simple rules for the cases you know about (it says 'same vendor', but there's no vendor_id in the args), plus a small LLM judge on a sample. Drift is often the first visible sign of a bad run, as it was in the incident above.
• ▹Context regressions. After each deploy, replay a fixed set of inputs and diff the context_refs for each decision. If the same question now pulls different chunks, or skips a tool output it used to see, then retrieval or prompting has changed, even when the final answers still look fine.
• ▹Human overrides become eval labels. When a user retries, edits the output or rejects an action, or a reviewer reverses it, write that into human_signal on the decision that caused it. You end up with labelled, real-world failures tied to exact decisions. You won't get better eval data than that.

Honest limits

• ▹Self-reported reasons are not ground truth. Models rationalise. A stated reason can sound perfectly coherent and still have nothing to do with what drove the output. Treat stated_intent as a claim to check against the actions and context_refs, never as the explanation. The useful signal is usually where what it said and what it did disagree.
• ▹The intent field costs tokens and can nudge behaviour slightly. Keep it to one sentence, and run evals before and after adding it, as you would for any prompt change.
• ▹PII and cost. Context by reference helps with both, but the context store still holds sensitive content. Mask personal and financial fields before you hash and store anything. Keep access to the raw prompt store separate from access to the decision table. Set retention rules: short for raw prompts, longer for decision records, which carry far less sensitive data.
• ▹It doesn't replace APM. You still need latency, error and cost metrics. Decision records sit on top of them, alongside standards like the OpenTelemetry GenAI semantic conventions.

Monday checklist

• ▹Wrap your tool router so every tool call, route and final answer passes through one function that writes a decision record. One choke point is far easier to maintain than logging scattered across the codebase.
• ▹Make intent and reason required fields on every tool schema, and strip them before execution.
• ▹Hash every retrieved chunk and tool output, store each one once, and log only the refs. Tag the refs in the prompt so the model can cite them.
• ▹Thread parent_decision_id through your agent loop, so each decision points to the one whose output it consumed.
• ▹Write the 'walk back from bad output' query (the recursive CTE above) and save it. Practise it on a known bad run now, before you need it under pressure.

Where this sits in the series

On Days 26 and 27 you built offline evals. They tell you whether the system is good on average across a dataset. Decision records tell you why one specific production run went wrong. You need both: evals catch regressions before you ship, and decision records explain the failures your evals didn't predict. Day 29 closes the loop. Today's human_signal labels and drift flags become the input to a production feedback loop that feeds real failures back into your eval sets and prompts.

Extend your knowledge

• ▹Read Michael Nygard's original 'Documenting Architecture Decisions' post. Put its structure (context, decision, status, consequences) side by side with the schema above.
• ▹Go through the OpenTelemetry GenAI semantic conventions and work out where decision fields could live in your current tracing setup, either as span attributes or as linked events.
• ▹Try an LLM observability tool such as Langfuse, LangSmith or Arize Phoenix. Check whether you can attach custom decision metadata and user feedback to individual steps rather than only to whole traces.
• ▹Read Charity Majors on observability vs monitoring (high-cardinality, queryable events). Decision records take the same idea and apply it to model choices.