Prompt Caching Cut Your Tokens 90% — So Why Is the Bill the Same?

Day 3: You Cached the Wrong Layer

You flip on prompt caching, watch the "cached tokens" line on the dashboard plummet, and feel good about it for exactly one billing cycle — until the invoice lands and it's basically the same number as last month. Nothing's broken. You just cached the layer that was never costing you anything.

The bill that didn't move

Picture the invoice, before and after. The "input tokens (cached)" line drops — 80, sometimes 90 percent. Nice. Except that line was already the cheap part of the bill. Meanwhile the tool-call and external-API line items — the ones that never show up in your model provider's dashboard because they're not technically an LLM cost — sit exactly where they were. Total spend: flat. I keep seeing this on teams that treated "enable prompt caching" as the whole caching strategy instead of one line item in it.

Where Day 1-2 left off

Days 1 and 2 were about what NOT to build yourself in an agent stack — orchestration, memory, guardrails. Most of that is buy-or-adopt, not build. Today's question sits one level deeper: given that a caching layer exists, where in the pipeline does it actually belong? That's a placement decision. It's solution architecture, not a code tweak you sneak into a PR.

The trace walkthrough

Pull up a real trace from a multi-agent run — planning, tool use, re-planning, the works. Split the cost into two buckets and look at the shape, not the exact numbers. The token cost is already cached: small, mostly system prompt and tool schemas repeated every turn. Sitting right next to it is the tool-call cost — uncached, and it dominates. The redundant calls almost always trace back to the same three culprits:

• ▹A DB lookup for the same entity ID, re-fetched every time the planner loops back around, because the planner has no idea it already has the answer
• ▹A retrieval or vector search re-run with the exact same query after a retry, because the retry logic re-executes the whole step instead of resuming from where it left off
• ▹An external API call — pricing, inventory, some third-party service — hit again on every re-plan, because the agent's "context" is the conversation history, not a memoized result

None of that shows up as "LLM cost." It shows up as latency, and as a bill from whatever sits behind the tool. On a pipeline where re-planning fires a handful of times per task — not unusual once you add retries or self-correction — that's roughly that many multiples of the tool cost for almost no extra token cost. The prompt cache is doing exactly its job. It's just doing it to the part of the system that was never the bottleneck.

Why prompt caching can't touch this

This isn't a bug in prompt caching. It's a scope limit you need to internalize. Prompt caching in an LLM API works by deduping the prompt prefix — system prompt, tool definitions, earlier turns — so the model doesn't reprocess, and you don't repay for, tokens it's already seen in that same structural position. That's the whole mechanism. It has zero visibility into what a tool did with the arguments the model generated. Re-plan, call the same tool with the same arguments again, and that's a brand-new tool invocation as far as the cache is concerned — the prompt around it might be identical and hit the cache, but the tool call inside it fires anyway. The cache dedups the ask. Not the work.

The architectural fix: a tool/retrieval result cache

The fix is a second cache, at a different layer, keyed differently. Prompt caching lives inside the LLM API call and keys on the conversation prefix. A tool/retrieval result cache sits between your agent's tool-calling layer and the actual DB, vector store, or external API — and it keys on (tool name, normalized arguments), full stop. The conversation doesn't matter to it at all. Same tool, same args, anywhere in the trajectory, any agent, any re-plan: cache hit, no re-fetch.

This is a solution-architecture call, not something you bolt on after profiling a slow demo, for three reasons: it changes where state lives (the cache now owns a slice of truth your agents depend on), it changes your consistency guarantees (more on that below), and it changes your service boundaries (the cache sits as its own component in front of tool executors, not tucked inside one agent). You decide this when you draw the pipeline — the same way you decide where a queue or a rate limiter goes, not after the demo runs slow.

The catch — teaser for Day 4

Before you go build this: a naive tool-result cache is dangerous the second a call is non-idempotent or context-sensitive — a payment call, a "get current inventory" call where staleness actually matters, a search whose right answer depends on something outside the args you're keying on. Cache the wrong call and your agent acts on data that was true five re-plans ago. That's tomorrow: cache invalidation in agent trajectories. Not solving it today. Just flagging it so you don't ship this blind.

One thing to check today

Grep your agent logs for duplicate tool calls with identical arguments inside a single trajectory — before you touch a single caching config. Find them, and you've found where your money actually goes.

Extend your knowledge

• ▹Instrument your traces to log (tool_name, args_hash) per call, then query for duplicates within a single trajectory — it's the cheapest way to size this before you build anything
• ▹Read your LLM provider's prompt caching docs closely for exactly what gets cached — usually prefix-based, with a minimum token threshold and a TTL — so you know precisely where the boundary stops
• ▹Check how your orchestration framework (LangGraph, CrewAI, or your own hand-rolled planner) exposes tool-call interception — that's the hook point for a result cache, not the LLM client
• ▹Design the invalidation policy — TTL per tool, explicit bust on write, or idempotency-key based — before you ship the cache. Day 4 goes deeper here