Why 'Temperature=0' Didn't Stop Your Agent From Changing Its Mind

Day 7: Temperature=0 Doesn't Mean Deterministic

Here's the rule everyone repeats: set temperature=0 and you get deterministic output. Here's what actually happens once you wire that into a real agentic pipeline: same prompt, same model, same everything you can see — and two runs come back with opposite verdicts. That gap between the rule of thumb and what actually gets logged is the whole lesson, and the sampler isn't where the bug is hiding.

The hook: same PR, same everything, different verdict

Take an AI code reviewer, point it at one PR, run it twice — same model, same prompt template, temperature locked at 0. Run one: 'approve, minor nit.' Run two: 'block, missing null check.' Nothing about the request changed. Or so it looked. That's the kind of result that should make you nervous, because it snaps the mental model most people bring into agentic engineering: temperature=0 equals deterministic, end of story. It isn't.

What temperature=0 actually promises

Temperature is a sampling knob. It reshapes the probability distribution the model draws its next token from. Set it to 0 and decoding goes greedy — at every step, the model just takes the single highest-probability token, no randomness involved. That's the entire promise. It's a claim about the sampler, not a claim about the world of data feeding tokens into that sampler.

The leap everyone makes next is seductive: greedy decoding is a pure function of its input, so the same input always produces the same output, so temperature=0 must mean determinism. True — but only under one condition: the input, the entire context window, has to be identical byte-for-byte across runs. In an agentic pipeline, that condition breaks quietly, and constantly.

The chase: ruling out a model bug

The first instinct when you hit this is to blame the model or the API: maybe there's batching noise server-side, maybe 'temperature=0' isn't perfectly greedy under the hood, maybe it's non-determinism baked into how the provider executes kernels (real, but rarely the dominant effect at the scale most teams run at). Before chasing any of that, log the exact prompt sent to the model on both runs and diff them. Do that here and the diff isn't empty — the file order in the 'here's the PR diff' section flipped between the two runs. Same files, same content, different sequence. The retrieval step that assembled the diff had reordered the files non-deterministically.

The real culprit lineup

None of the items below touch the sampler. Every one of them touches the context window — the actual input that 'deterministic' greedy decoding operates on:

• ▹Unordered tool results — a file-listing or search tool backed by a set, hash map, or parallel fetch hands back results in whatever order they happened to resolve, not a stable one
• ▹Wall-clock timestamps baked into prompts — 'current time: ...' or 'as of ...' headers that agent frameworks stamp in automatically for logging or freshness, without you asking
• ▹A non-stable sort in the diff assembler — stable on ties in theory, but fed pre-shuffled input, or using a comparator that never fully orders equal-priority items
• ▹Race conditions across parallel tool calls — two tools fire at once, and whichever one returns first gets slotted first into the prompt
• ▹Ambient IDs — request IDs, trace IDs, UUIDs sitting in tool outputs that change every run and end up concatenated straight into the context

The reframe: determinism belongs to the whole pipeline

Determinism isn't something you buy from a decoding parameter. It's a property of the entire context window your pipeline assembles — retrieval, tool orchestration, formatting, timestamps, ordering — with temperature as just one input among many. Greedy decoding is deterministic given a fixed input. Your pipeline is the thing responsible for actually fixing that input, and most agentic pipelines never once audit whether it does.

Checkpoint: connecting back to what temperature is

If you want the sampling-math version of temperature — how it reshapes the softmax distribution, why temperature=0 is the limiting case of that reshaping, how it differs from top-k or top-p sampling — that's the technical definition sitting underneath this lesson, and it stands on its own. What this lesson adds is the systems-level failure mode: knowing the math doesn't save you if you never stopped to ask what else changes between two runs of your pipeline.

Practical takeaway

If you genuinely need reproducible agent behavior — for regression-testing an agent, for auditing a decision, for figuring out 'why did it do that' — pin and log the full context, not just the sampling parameters:

• ▹Sort everything with an explicit, stable key before it ever enters the prompt — file paths, tool results, retrieved chunks
• ▹Strip or freeze timestamps and request IDs from anything getting concatenated into the prompt, unless the task genuinely depends on freshness
• ▹Log the exact final prompt string for every run, not just the inputs to your pipeline — diff prompts across runs the moment behavior diverges, don't guess
• ▹Treat temperature=0 as necessary and nowhere near sufficient for reproducibility — it removes exactly one variable, not the rest of them

Tomorrow's thread

Pinning the full context works, until it doesn't. Some sources of variance — live web results, concurrent multi-agent state, wall-clock-dependent tools — can't be frozen without breaking the whole point of the agent. Tomorrow: what actually varies when you can't pin the context, and how to design agent evaluation and monitoring around living with non-determinism instead of pretending it away.

Extend your knowledge

• ▹Read up on softmax temperature scaling and top-k/top-p sampling to get the math underneath greedy decoding — Anthropic's and OpenAI's API docs cover temperature, top-k, and top-p as parameters, but the softmax math itself is easier to find in a general ML explainer than in either provider's reference
• ▹Audit one of your own agent pipelines: log the full assembled prompt for two runs on identical input and diff them — odds are you'll find at least one non-deterministic ordering or timestamp hiding in there
• ▹Look at how eval frameworks for LLM agents handle non-determinism — most don't try to eliminate it, they design assertions that tolerate it
• ▹Tomorrow's Day 8 lesson: living with non-determinism in agent pipelines instead of pretending it away