The Agent Kept Insisting It Was March. The Cache Was Just Doing Its Job.

Day 12 — Idempotency Breaks When the Function Isn't Stable

An idempotency key has one job: stop a retry from double-charging a customer or double-writing a row. That job only works if the thing sitting behind the key never changes. Put an LLM call in your pipeline and that assumption quietly stops being true — the function now ships new versions on your normal deploy cadence, and a cache that doesn't know that will defend a stale answer with exactly the same confidence it defends a correct one.

The ticket: "why does the agent still think it's March"

A support ticket lands: a scheduling agent is calmly, repeatedly telling users the current month is March. It's September. First guess is a prompt bug in the model that shipped last week — that's usually where this kind of thing lives. Two hours of tracing later, the new model isn't the problem at all. An idempotent cache stage upstream never invalidated. It re-served a response generated weeks earlier, before the model swap, because the input hash still matched. The "bug" is the safety mechanism doing precisely what it was built to do, aimed at an assumption that had quietly stopped being true.

What idempotency actually promises

In a classical pipeline, an idempotency key buys you exactly three guarantees — no more:

• ▹Same input → same output, so retries are safe to replay
• ▹No duplicate side effects — a payment fires once even if the request hits three times
• ▹A stable identity for "have I already done this work?" checks

All three lean on one quiet assumption underneath: the function mapping input to output never changes. That holds for "charge this card" or "insert this row" — that logic doesn't get retrained overnight. It does not hold for "ask the LLM to summarize this ticket," because the function itself — model weights, system prompt, tool schema — ships new versions on a completely ordinary engineering cadence.

Where the assumption breaks: the LLM call has more inputs than the prompt

A classical idempotency key is usually just hash(request_payload). Teams lift that pattern straight onto an LLM stage without noticing they've dropped most of the function's real arguments on the floor:

LLM_output = f(user_prompt, model_id, system_prompt, tool_definitions, temperature, ...)

# what actually goes in the cache key:
cache_key = hash(user_prompt)   # <- only 1 of ~5 real inputs

Ship a new model, tweak the system prompt, add a tool definition — none of it shows up in the key. The cache layer can't tell "same request, safe to reuse the answer" apart from "same request, but the function answering it has moved on." It picks the first case every time, because that's the only one anyone built it to handle.

Why your test suite never caught it

Tests retry the same input against the same deployed version of everything — model pinned in CI, prompt file checked into the repo, tool schema baked into the fixture. Staleness only exists across a deploy boundary, and nothing in a normal test run crosses one. The bug just sits there, dormant, until someone actually ships a new model or prompt weeks later — at which point it reads like a regression in the new version, not the caching defect that's been latent since day one.

The fix: version-scoped cache keys + deploy-triggered invalidation

Two changes, and they only work shipped together. First, the key has to encode every dimension that can change the output, not just the payload. Second, expiry has to fire on a deploy event, not a wall-clock TTL — a TTL just delays the bug, it doesn't fix it.

cache_key = hash(
    user_prompt,
    model_id,          # e.g. "claude-sonnet-5-2026-01"
    prompt_version,     # git sha or semver of the prompt template
    tool_schema_hash,   # hash of the tool/function definitions
)

# invalidation: not TTL(24h), but on deploy
on_deploy(new_model_id | new_prompt_version | new_tool_schema):
    bump_cache_namespace()   # old keys become unreachable, not just stale

Bumping a namespace on deploy is cheaper than hunting down and deleting old keys one by one, and it buys you something a TTL can't: the cache can never silently serve an answer generated by a version that no longer exists in production.

The generalizable lesson for Day 13

Any "idempotent" step wrapping a non-stable function needs a key that fingerprints the whole function, not just the request. Before you trust a cache anywhere in an agentic pipeline, run it against this checklist:

• ▹Does the key include the model/provider ID, not just the prompt text?
• ▹Does the key include a version for the system prompt and any few-shot examples?
• ▹Does the key include a hash of tool/function definitions passed to the model?
• ▹Is invalidation tied to a deploy event, or just a TTL that hopes nothing changes in the meantime?
• ▹If you rolled back the model tomorrow, would the cache serve rolled-back-correct answers, or a mix of both versions?

Extend your knowledge

• ▹Audit one existing LLM-call cache or memoization layer in your codebase against the Day 12 checklist above — check whether model_id and prompt_version actually appear in the key.
• ▹Read Anthropic's docs on prompt caching to see how they scope cache validity (system prompt + tools + messages prefix) — it's the same versioned-function problem solved at the provider layer.
• ▹Look at how your deploy pipeline exposes a model/prompt version identifier at runtime — if there's no such identifier available to your code, that's the actual gap to fix before touching the cache key.
• ▹Think about the same failure mode one layer up: does your embedding cache for RAG retrieval encode the embedding model version, or just the input text?