The JSON Was Valid. The Account ID Was Fake. Nobody Noticed for Days.

Day 2: Structured Output Guarantees Shape, Not Truth

Your pipeline logged 100% valid JSON all week. Feel good about that? Don't. Schema validation checks that a response is shaped correctly — it has zero opinion on whether the values inside it are true. Every function-calling schema, every Pydantic model, every JSON mode setup you've wired up proves shape, not truth. That gap between 'parses' and 'correct' is exactly where production incidents go to hide.

The incident

A PhoenixDX pipeline step called an LLM to pull structured metadata out of customer tickets — category, priority, account_id. One response came back clean: every field present, every type correct, validator lit up green. Except account_id was invented — a plausible-looking ID that matched no real account. Nothing crashed. No exception fired. The next agent in the chain joined on that ID, wrote to the wrong record, and nobody caught it for days — because a hallucination wrapped in valid JSON looks exactly like success.

The trap, in one line

Structured output smashes two separate guarantees into one green checkmark: 'is this shaped right' and 'is this true.' Schema validation only ever answers the first one. Teams read the checkmark as an answer to the second anyway.

Why this happens: the mechanism

Here's the mechanism. Constrained decoding — grammar-based sampling, schema-enforced JSON modes like OpenAI's Structured Outputs, forced tool_choice — works by masking the token distribution at every step, so the model can only emit tokens that keep it on a valid path through your schema. That kills syntax errors. It also means every required field gets filled, even when the model has nothing real to fill it with. In free text, uncertainty has somewhere to go: 'I think the account might be...', 'unclear from context,' a hedge, a flat refusal. Constrained decoding removes that exit. A required field can't sit empty and can't hedge — it has to be filled, so the model reaches for the most statistically plausible-looking value, signal or no signal. That's a hallucination wearing a valid schema.

Free text (uncertainty visible):
  "The account ID isn't clearly stated in the ticket, possibly ACC-4471?"

Constrained JSON (uncertainty erased):
  {"account_id": "ACC-4471", "category": "billing", "priority": "high"}
  ^ schema valid, type-correct, and wrong — with no signal left to tell you so

The tell: how this bug hides

• ▹It doesn't crash. It doesn't throw. The JSON parses clean, every single time.
• ▹It sails through every type check and every required-field check your validator runs.
• ▹It only surfaces downstream — once another agent, or a human, trusts the field and acts on it.
• ▹By the time anyone notices, the chain back to 'the LLM guessed' is several hops away and a pain to reconstruct.

A minimal fix pattern: two gates, not one

Split validation into two separate checks, and run them as two separate gates in your pipeline — not one combined step that quietly conflates them.

• ▹Gate 1 — Schema validator: does it parse, are types correct, are required fields present? (Pydantic, Zod, JSON Schema — pick your poison.) This catches malformed output. It says nothing about wrong output.
• ▹Gate 2 — Content validator: is the value actually plausible given the input? Rule-based (does account_id exist in the source ticket text or your DB?), cross-field (does priority=high make sense given category=billing and zero escalation keywords?), or a second LLM-as-judge pass that re-reads the source and the extracted field and scores agreement.
• ▹Log which gate rejected what. A schema failure and a content failure are different bugs with different fixes — collapse them into one 'validation failed' log line and you've killed your own ability to debug either.

This is exactly why Day 3 has to be about verification and repair loops. Structured output was never the safety net — it just looks like one, because a green checkmark and a value you can actually trust produce the identical visual signal in your logs. The real safety net is gate two, plus a loop that catches a content failure and retries, repairs, or escalates it — instead of shipping it downstream and finding out three days later.

Closing thought: 'valid' is a syntax word, not a trust word. Stop grading your agents on it.

Extend your knowledge

• ▹Read your LLM provider's structured-output / tool-use docs closely — look for language that separates 'guaranteed valid JSON' from 'guaranteed correct values.' Most vendors say this explicitly if you go looking.
• ▹Add a confidence or source_span field to your schema — require the model to quote the exact source text it extracted a value from. Gives your content validator something concrete to check against.
• ▹Look at how libraries like Instructor or Guardrails implement post-validation hooks — they're built around exactly this two-gate pattern.
• ▹Prototype an LLM-as-judge content gate on one existing pipeline step this week: same input, second pass asking 'does this extracted field actually match the source?' See how often it disagrees with your schema-only validator. You probably won't like the number.