Prompt caching for coding agents

One line in a system prompt can make your coding agent several times more expensive, and nothing will tell you. Tests still pass and diffs look the same. Reviewers don't notice a thing. The invoice goes up, and the pause before the first token gets a little longer. In the case I keep running into, someone added that line months ago because the model kept getting the date wrong: `Current time: 14:32:07`, right at the top of the system prompt.

Before you read any further, run this against your harness:

grep -rnE 'now\(|datetime|Date\.now|new Date\(|time\.time|strftime|git status|rev-parse|uuid|nanoid|random' \
  --include='*.py' --include='*.ts' --include='*.js' \
  path/to/your/prompt-building/code

Look at where the hits land. If any of them are in code that builds the system prompt, the tool list, or anything else that renders above the conversation, you're probably paying full price on every turn. Keep reading.

The 90-second mental model

Prompt caching matches an exact prefix, byte for byte, from the first token. It isn't semantic and it doesn't fuzzy-match. Say the first 40,000 tokens of this request are identical to the first 40,000 of a recent request, and you put a cache breakpoint there. Those tokens get read from cache at a fraction of the normal input price, and most of the prefill work is skipped. Matching stops at the first byte that differs. Everything after it is processed again at full price. If there's a breakpoint further down, you also pay a premium to write a new cache entry.

That makes render order the most important property of your prompt. The Anthropic API fixes the top-level order: tools, then the system prompt, then messages. Within each of those, the layout is up to you. A typical coding agent request looks like this:

If you drop a per-second value into layer 2, layers 2 through 5 miss the cache on every request. The tool definitions above it can still hit, but only if you put a breakpoint right after them. In most harnesses the history holds most of the tokens anyway, so a cached tool list doesn't help much.

Step 1: Measure before you touch anything

Every response already tells you what happened. The `usage` object has three input fields, and you should log all three on every call:

• ▹`cache_read_input_tokens`: served from cache. Cheap and fast.
• ▹`cache_creation_input_tokens`: written to a new cache entry and billed above the base input price.
• ▹`input_tokens`: the uncached tokens after the last breakpoint, at the normal price.

def log_cache(resp, session_id, turn):
    u = resp.usage
    read = u.cache_read_input_tokens or 0
    write = u.cache_creation_input_tokens or 0
    fresh = u.input_tokens
    total = read + write + fresh
    hit_ratio = read / total if total else 0.0
    metrics.emit("agent.cache", {
        "session": session_id, "turn": turn,
        "read": read, "write": write, "fresh": fresh,
        "hit_ratio": round(hit_ratio, 3),
    })

Take a handful of real sessions and plot `hit_ratio` against turn number. A healthy loop starts low on turn 1, when at most the shared stable layers are already cached. It jumps on turn 2 and stays high from then on, because each turn only adds the newest message and tool output. The shape tells you what's wrong:

• ▹A sawtooth means something resets the prefix every few turns. Compaction, a tool reload and a CLAUDE.md write are the usual causes.
• ▹A flat line near zero means a volatile value sits above every breakpoint, or there are no breakpoints.
• ▹Also chart `write` by itself. If it's large on every turn, you're paying the write premium over and over and never getting the reads back.

Keep these graphs. Check every change below with a before/after on the same graph. Otherwise you're guessing.

Step 2: Sort your context by how often it changes

List everything that goes into a request and write down how often each piece actually changes. Check the code for this. Don't go by how often you think it changes.

Element                         Change frequency
------------------------------  ----------------
Tool schemas (built-in)         per release
System prompt (role, rules)     per release
MCP tool schemas                per session (if pinned)
CLAUDE.md / project conventions per session (if frozen)
Repo map / architecture notes   per session
Conversation history            append per turn
Git branch, git status          per turn
Open files, cursor, diagnostics per turn
Latest tool output              per turn
Current date                    per day
Current time                    per second

Sort the table by change frequency and you have the order your prompt should render in. The rule is short: nothing that changes per turn or per second may sit above a cache breakpoint you depend on. When I did this for one of our internal harnesses, I found three per-turn items in the system prompt. Nobody had put them there on purpose. Three different people had each made a reasonable "give the model more context" change.

Step 3: Fix the timestamp class of bugs

There are two fixes, and you usually want both. First, cut the precision. The model almost never needs seconds, and a date stays the same for 24 hours. Second, move the value below the cached prefix: into the newest user turn, or into a small context block appended after history. Git status, branch name, open-file lists, lint diagnostics and the "current task" banner need the same fix.

 def build_request(session, history, user_msg):
-    system = f"Current time: {datetime.now():%H:%M:%S}\n"
-    system += f"Branch: {git_branch()}\n{git_status()}\n"
-    system += BASE_SYSTEM_PROMPT + session.claude_md
+    system = [
+        {"type": "text", "text": BASE_SYSTEM_PROMPT},
+        {"type": "text", "text": session.claude_md_frozen,
+         "cache_control": {"type": "ephemeral"}},
+    ]
+    live = f"<env>date: {date.today()}\nbranch: {git_branch()}\n{git_status()}</env>"
-    return system, history + [user(user_msg)]
+    return system, history + [user(live + "\n\n" + user_msg)]

There's a side effect. Once `live` is inside a user message, it stays in history. That's fine, because history is append-only, and it actually helps: the model can see how the branch state changed from turn to turn. Just never go back and edit the old `<env>` blocks.

Step 4: Freeze the tool list like an ABI

Tools render before the system prompt. Any change to a tool invalidates everything after it: your system prompt, CLAUDE.md and the whole conversation. Adding a tool counts. So do removing one, rewording a description and reordering properties. I treat the tool list the way I'd treat a public ABI:

• ▹Sort tools by name before serialising, every time.
• ▹Serialise JSON with a fixed key order. Look for `json.dumps` without `sort_keys=True` over dicts built at runtime, and for any loop over a `set` of tool names in that path.
• ▹Pin MCP server versions for the length of a session. Suppose an MCP server upgrades itself mid-session and adds one sentence to a tool description. Every agent connected to it now pays for a full cache rewrite.
• ▹Don't hot-load tools mid-session. If an agent needs a new capability, start a new session or route it through a stable dispatcher tool.

Here's the trade-off. Many teams build "smart" tool selection that sends only the 8 tools relevant to this turn instead of all 40. It looks like a token saving. But if the selected set changes between turns, each change is a cache miss on everything downstream. A large, stable tool list that gets cache-read every turn can cost less than a small one that keeps changing. I don't think one answer fits every team. Run both setups, compare them on your Step 1 graphs, and go with what the numbers say.

Step 5: Treat CLAUDE.md as a cache contract

Multi-agent setups get hurt most by shared CLAUDE.md files. Picture four parallel agents on one repo, all reading the same CLAUDE.md into their prefix. One agent learns something useful and appends a line. A teammate fixes a typo. Every agent that re-reads the file on its next turn now has a different prefix from that point down, and all four rebuild their caches.

• ▹Snapshot CLAUDE.md when the session starts and use that snapshot until it ends. Don't re-read the file every turn.
• ▹Batch edits at session boundaries. A change made mid-session takes effect in the next session, not this one.
• ▹Put new learnings somewhere else first: a memory block appended at the end of the context, or a scratch file the agent reads through a tool call. Promote them into CLAUDE.md deliberately, during review.

This also helps with governance. CLAUDE.md changes go through PR review, instead of whichever agent wrote to the file last.

Step 6: History is append-only by default

Every "tidy the context" feature works against caching. Summarising older turns, trimming stale tool output and rewriting an earlier message to remove a mistake all change bytes in the middle of the prefix. Everything after the edit point drops out of the warm cache. You end up with fewer tokens, but more of them are uncached.

My rule: compact only when the projected savings over the remaining turns outweigh one full cache rewrite plus the cache hits you lose. Roughly:

Cached reads are cheap, so cutting tokens that would have been cache hits saves less than it looks like. In practice, compaction should be rare and deliberate. Do it at natural checkpoints like the end of a subtask or when you hit the context-window limit. Don't run it as a background job that trims old tool output every few turns. If you have one of those, check your hit-ratio graph. It's probably the sawtooth.

Step 7: Place breakpoints on purpose

The API gives you a small number of explicit `cache_control` breakpoints per request, currently four. For a coding agent, this is where I'd put them:

• ▹After tools + system prompt. This is the most stable layer, shared by every session on the same harness version. That only holds within one workspace, because caches aren't shared across workspaces or organisations.
• ▹After the stable project context, meaning the frozen CLAUDE.md and repo map. Parallel agents on the same repo and snapshot share this one.
• ▹On the last message block, at the moving edge of the conversation. Each turn reads everything before it and writes only the new tail.
• ▹Save the fourth for a large artifact that stays stable mid-session, such as a spec document the agent keeps referring back to.

Then there's lifetime. By default, a cache entry expires a few minutes after it was last used, and every hit refreshes it. A tight agent loop stays warm without any effort. Idle agents and human review loops don't. If your workflow is "agent proposes, human reviews for ten minutes, agent continues," the next turn probably starts cold. You have two options. One is the longer-lived cache option for the stable layers; the write premium is higher, so check current pricing and measure. The other is to accept the cold start and keep the stable layers large and shared, so someone else's session keeps them warm. Your Step 1 graphs will tell you which situation you're in. If the hit ratio dips after every long gap, the problem is TTL, not ordering.

The cache-bust checklist for PR review

Add this to your PR template for any change that touches prompt assembly, tool registration or context files:

• ▹[ ] No wall-clock values (`now()`, `Date.now`, `time.time`, `strftime`) in the system prompt or tool definitions. Dates go into the user turn.
• ▹[ ] Tool list is sorted, serialised with fixed key order, and never built by iterating a `set` or an unordered dict.
• ▹[ ] No per-request IDs, nonces, trace IDs or random seeds anywhere above the last breakpoint.
• ▹[ ] No dynamic tool registration or MCP server reconnects mid-session. MCP versions are pinned per session.
• ▹[ ] Agents can't write to CLAUDE.md during a session. Learnings go to an appended memory block.
• ▹[ ] No code path rewrites, reorders or trims earlier messages, except an explicit, logged compaction step.
• ▹[ ] A/B prompt variants and feature flags are injected below the cached prefix or fixed per session, never at the top.
• ▹[ ] Locale, user name, timezone and other per-user text sit after the shared prefix, not inside it.
• ▹[ ] The PR description includes hit-ratio before/after from at least one real session.

Make the cache visible

Put the hit ratio on your agent dashboard, next to p50 time-to-first-token and cost per session. Alert when the rolling hit ratio drops below the baseline you measured in Step 1. Caching never fails loudly. The prompt is still valid and the model still answers well. The only symptom is a bill that grows a little every week. None of your other signals will catch it, so you need a metric that does.