JSON Mode Never Throws an Error — That's Exactly Why It's Dangerous

Day 3 — The Hidden Cost of JSON Mode: When Structured Output Kills Reasoning

Turn on JSON mode and every response comes back parseable. Zero errors, zero retries, a clean green dashboard. And your accuracy can be quietly getting worse the whole time — because forcing an LLM into a schema from the very first token doesn't just format the answer, it changes how the model thinks. On planning, multi-hop QA, agent tool routing: this failure is real, and it hides in plain sight, because 'valid JSON' and 'correct answer' are not the same signal — and most of us only check the first one.

Valid JSON every time, wrong answers anyway

Here's the setup, and you've probably shipped a version of it. You turn on JSON mode — or a strict schema through function calling — for something like: given these support tickets, decide the root cause category and the recommended fix. Every response comes back syntactically perfect. No exceptions, no retries, no parse errors, ever. Then somebody finally checks the actual answers against ground truth, and accuracy has dropped compared to the old free-text prompt. Nothing crashed. Nothing logged. The regression was invisible the whole time because you were watching the wrong signal: 'did it parse,' not 'was it right.'

What actually changed under the hood

JSON mode and schema-constrained function calling — OpenAI's Structured Outputs, Anthropic's tool_use with a forced tool choice, grammar-based libraries like Outlines or llama.cpp's GBNF grammars — all do the same trick under the hood: they constrain decoding itself. At every generation step, the sampler masks out any token that doesn't fit the next valid slot in the schema's grammar. That's the whole mechanism behind the 100%-valid-JSON guarantee. The model isn't being well-behaved — it's structurally incapable of emitting anything else.

And here's the cost: the very first token has to already be part of the answer shape — brace, then key, then value. There's no room left for 'let me work through this step by step' unless you carve out a field for it on purpose. This matters because autoregressive models reason by writing. Each token conditions the next, so free-text chain-of-thought isn't decoration — it's the mechanism the model uses to incrementally build toward a correct answer. Skip straight to the answer slot and you've skipped the computation that used to happen in the words leading up to it.

Side-by-side: same prompt, same model, two pipelines

Task: 'A customer says their invoice is $40 higher than quoted. The quote was $210/month for the Pro plan plus one add-on seat at $15/month. They were charged $265. Identify the discrepancy and the corrected amount.'

// Pipeline A — free text, then extract
Prompt: "Work through this step by step, then give the final answer."

Model output:
"Quoted total: $210 base + $15 add-on = $225/month.
Charged: $265.
Difference: $265 - $225 = $40.
Checking plan tiers: Pro plan add-on seats are billed at $15,
but there's also a $40 'legacy proration' line typical of mid-cycle
upgrades — that matches the discrepancy exactly.
Root cause: mid-cycle upgrade proration, not a billing error.
Corrected recurring amount going forward: $225/month."

// A second, cheap call extracts the final two facts into JSON.
{ "root_cause": "mid-cycle proration", "corrected_amount": 225 }

// Pipeline B — JSON mode from token one
Prompt + schema: { root_cause: string, corrected_amount: number }

Model output (valid JSON, wrong):
{ "root_cause": "billing error", "corrected_amount": 250 }

// No arithmetic was ever written down. No hypothesis about
// proration was ever formed. The model had to commit to a
// root_cause string as its literal first content tokens, before
// it had 'seen' the numbers resolve on the page.

Same weights. Same prompt intent. Materially different accuracy. Pipeline B isn't a dumber model — it's the exact same model with its scratch space taken away. It skipped straight to pattern-matching a plausible-sounding label instead of actually deriving one.

Why this is invisible in testing

Most pipeline health checks test parseability, not correctness — did the call throw, did json.loads succeed, does the shape match the schema. JSON mode makes every one of those checks pass by construction. That's not a bug, that's the entire point of the feature. So if your eval suite is a schema validator instead of a grader that compares the extracted value against ground truth, you will never catch this regression. It's the same failure mode as an agent framework reporting 'tool call succeeded' when what it actually means is 'the API returned 200' — not 'the agent did the right thing.' Green checkmarks measure the wrapper. They say nothing about the reasoning inside it.

The fix pattern: reason-then-format

The fix is simple to state: give the model somewhere to think before it has to commit to the schema. Two ways to do it, in order of preference.

// Option 1 (cleanest): two calls
// Call 1 — no schema, just ask for reasoning in free text
// Call 2 — cheap/small model extracts the final answer into
//          your schema from call 1's output

// Option 2: one call, but put reasoning INSIDE the schema,
// as a field that comes BEFORE the answer field
{
  "reasoning": "string field first",
  "root_cause": "string field second",
  "corrected_amount": "number field third"
}
// Field order matters: the grammar still generates left to right,
// so 'reasoning' must be the first key the model is allowed to
// fill, giving it token budget to think before answer fields lock in.

Option 2 is the one everyone reaches for, because it's a single API call and it does work — but only if the reasoning field genuinely comes first and actually gets room to breathe. Put 'answer' before 'reasoning' in your schema and you've just recreated the original bug with extra steps.

Decision rule: when JSON mode is safe vs. when it costs you

• ▹Safe — single-step classification or extraction: a sentiment label, a named-entity pull, 'which of these five categories.' The answer is a lookup, not a derived conclusion.
• ▹Safe — routing to one of a fixed set of tools or agents in a multi-agent system, where the decision itself is shallow even if what happens downstream is complex.
• ▹Risky — anything needing arithmetic, multi-hop lookup, planning, or weighing several pieces of evidence before concluding. That's exactly the shape of the 'plan the next step' call in a ReAct-style agent loop.
• ▹Risky — LLM-as-judge evals. If the judge has to weigh multiple criteria before scoring, force a bare verdict field with nothing before it and you'll get noisier, less defensible scores.
• ▹Rule of thumb — if a human would reach for a scratch pad to get this right, the model needs one too. Give it a reasoning field, or a free-text first pass, before you lock it into a schema.

Where this fits next

Once you're deliberately producing a reasoning trace ahead of the structured answer, the next problem shows up at the system boundary: validating that final object against your contract, versioning the contract as it evolves, and deciding what happens the moment validation fails. That's Day 4.

Extend your knowledge

• ▹Read OpenAI's Structured Outputs docs on how their constrained decoding grammar actually works, then compare it against Anthropic's forced tool_use behavior.
• ▹Go read the Outlines library (dottxt-ai) source for a concrete implementation of grammar-constrained sampling — it makes the token-masking mechanism tangible instead of abstract.
• ▹Check the Instructor library's docs and cookbook for the 'chain_of_thought' field pattern — a reasoning field placed ahead of the answer field — and see for yourself why the ordering is the whole point, not a detail.
• ▹Run your own A/B. Take one multi-step task from your own pipeline, run it both ways — JSON-mode-only and reason-then-extract — and grade both against ground truth, not schema validity.