Same Input, Different JSON — The Bug Wasn't in the Prompt

Day 8: The Sampling Parameter Nobody Set on Purpose

Same input, same prompt, same model — and the tool call comes back malformed one run in five. That's the moment most teams start rewriting the system prompt. Nine times out of ten, they're editing the wrong file. The actual bug is temperature and top_p, sitting untouched at whatever default the SDK shipped with, quietly deciding how much latitude the model has to wander off the grammar.

Three days of prompt rewrites that didn't fix anything

The demo had been clean. Tight tool calls, valid JSON, the same behavior every time the team ran it in front of stakeholders. Production told a different story: malformed arguments, a required field dropped here and there, valid JSON occasionally wrapped in a stray sentence the model felt like adding. So the team did what everyone does first — they went after the prompt. Tightened the instructions. Added "always respond with valid JSON." Threw in few-shot examples. Added a stern all-caps warning about schema compliance for good measure. Three days later, still flaky. Same input, sometimes clean, sometimes garbage. Nobody had touched the one parameter actually responsible.

The diff that actually mattered

It wasn't in the prompt file at all. It was sitting in the client call, in a parameter nobody had ever set — quietly inheriting whatever the SDK or framework defaulted to.

# Before — inherited defaults, nobody chose these
response = client.messages.create(
    model="claude-sonnet-5",
    messages=messages,
    tools=tools,
)

# After — the actual fix, three days late
response = client.messages.create(
    model="claude-sonnet-5",
    messages=messages,
    tools=tools,
    temperature=0.2,
    top_p=0.9,
)

Not a word of prompt text changed. The instructions were fine the whole time. What changed was how much freedom the model had when sampling from its own output distribution — and for a tool-calling agent, that distribution isn't just prose. It's braces, quotes, commas, argument tokens. The syntax lives in the same probability space as the words.

Why this is so easy to miss

Nobody catches this at demo scale, because nobody's sampling enough to see it. You run the agent five, ten, twenty times while building it, and any variance in there just reads as the normal noise you'd expect from a non-deterministic system — low stakes, low n, nothing that trips your instincts. Then it hits production volume. Thousands of calls a day. The exact same variance that was invisible at ten runs turns into a measurable failure rate at ten thousand. The default never got worse. You just finally sampled enough of it to see what was already there.

What temperature and top_p actually control

The model never just picks the single best next token. It computes a probability distribution over the whole vocabulary and samples from it. Temperature reshapes that distribution before the sample is drawn — turn it down and the distribution sharpens toward the highest-probability tokens; turn it up and it flattens, giving lower-probability tokens a real shot. Top_p, nucleus sampling, cuts it a different way: instead of reshaping the curve, it restricts sampling to the smallest set of tokens whose cumulative probability clears p, then samples inside that set.

For free-form prose, a wider distribution mostly buys you variety in phrasing — harmless, sometimes even the point. For structured or tool-call output, that same distribution includes the literal syntax tokens that make the output parseable: the opening brace, the closing quote, comma placement, field names, enum values. Widen it there and you're not making the model more "creative" — you're raising the odds it samples a token that looks plausible but is structurally wrong, at exactly the spot where a parser has zero tolerance for improvisation. The model isn't reasoning any differently between runs. It's sampling differently over a distribution that mixes content tokens with grammar tokens, and structured output doesn't forgive grammar drift.

The tell: reasoning failure vs. sampling artifact

Don't guess your way through this. Run the same input N times — 10 to 20 is plenty to see the pattern — and look at where the outputs actually diverge, not just whether they do.

• ▹If the content shifts but the shape holds — same JSON keys, same tool-call structure, just different values or phrasing — that's ordinary model variance or a real reasoning inconsistency. Go fix the prompt or the context, not the sampling.
• ▹If the shape itself is unstable — valid JSON on one run, truncated on the next, wrapped in explanatory text, a field renamed or missing entirely — that's a sampling artifact. The model is taking different paths through tokens it should never have had discretion over.
• ▹Quick way to confirm it: drop temperature to near zero for the same N runs. If the divergence collapses to near-identical output, you've just proven sampling was the variable — not the prompt, not the reasoning.

The fix, as a checklist

• ▹Pin temperature explicitly on every call path that produces structured output or tool calls. Never let it inherit the SDK or framework default.
• ▹Actually go look up what that default is. It's rarely front and center in the docs, and it differs across raw API clients, agent frameworks, and whatever orchestration layer wraps the call for you.
• ▹On tool-call-heavy paths, pin top_p too. Temperature and top_p interact — leaving one floating while you pin the other still leaves you exposed.
• ▹Use a tighter setting for anything a parser or schema validator is going to consume. Save the wider sampling for genuinely open-ended, prose-only generation.
• ▹Log the sampling parameters next to every trace — temperature, top_p, model version. When the flakiness comes back, you want that config sitting right next to the input/output, not buried three deploys back in history.

The broader lesson

Sampling is a production variable, same category as your timeout values and retry policies. You set those on purpose because you understand what happens if you leave them at default. Temperature and top_p deserve the exact same treatment, especially once tool calls and structured output are load-bearing behavior for your agent, not conversational flavor. Inheriting a default because it "felt fine in the demo" is precisely how a config-shaped bug gets misdiagnosed as a prompt-shaped bug and eats three days of your week. Tomorrow: why "just set temp=0" isn't the universal fix either — and the new failure mode you buy yourself by forcing determinism.

Extend your knowledge

• ▹Check the Anthropic API reference for the exact default values and valid ranges of temperature and top_p for the model you're running.
• ▹Audit your own agent framework's client wrapper (LangChain, your in-house orchestrator, whatever wraps the SDK call) — confirm what temperature/top_p it passes when you don't set them.
• ▹Run the N-run consistency test described above against your own tool-calling paths this week, before the next flaky-agent incident forces you to.
• ▹Preview tomorrow's lesson: why forcing temperature=0 isn't a universal fix, and what failure mode it introduces instead.