Prompt layout and TTFT

Day 29: Prompt layout decides your TTFT

In an agent loop, the slow part is often the wait before the model streams its first token, not the tool call. You pay that wait again on every turn. With prompt caching, how long it lasts depends mostly on the order of things in your prompt. The people who write the prompts control that order, not the infra team.

The hook: turn 1 vs turn 40

We had a coding agent that felt quick at the start of a session and slow by the end. The model, region and tools stayed the same. When we plotted time-to-first-token (TTFT) per turn, the line rose a little on every turn. It never spiked, so no alert fired. Then we added the cached-token count from the API response to the same chart, and the cause was obvious: almost nothing was hitting the cache. The prompt barely changed between turns, but the provider was processing it from scratch every time.

The cause was in our own harness. A helper put 'Current time: ...' near the top of the system prompt so the agent knew the date. That line changed on every request, so it broke the cache for everything after it, which was almost the whole prompt. The provider wasn't slow. Our code was.

What TTFT actually measures

TTFT is the time from sending the request to receiving the first output token. It has two main parts:

• ▹Queueing: time spent waiting for capacity on the provider's side. Apart from picking a tier and region, you can't do much about it.
• ▹Prefill: before the model can produce the first token, it reads your whole input and builds its internal state (the KV cache). The more input tokens you send, the more prefill work it does.
• ▹Decode (generating tokens) comes after prefill. It adds to total latency, but TTFT is mostly queueing plus prefill.

Agent loops make prefill worse because of how they work. Every turn re-sends the whole conversation: system prompt, tool definitions, every earlier tool call and result, and the new message. The input grows each turn. If none of it is cached, each turn prefills the full history again, and TTFT rises with the turn number.

Prompt caching in one picture

Prompt caching lets the provider reuse prefill work when the start of your prompt (the prefix) exactly matches one it has seen recently. Matching starts at the first token and stops at the first byte that differs. Everything after that byte is processed from scratch, even if the rest of the prompt is identical.

This builds on earlier days. On Day 'context windows' we treated the prompt as a token budget. On Day 'tool calling' we saw that tool definitions and tool results are just more tokens in that window. Caching adds one more rule: position matters. A token is cheap only if nothing before it has changed since the last call.

• ▹Anthropic: you mark cache breakpoints with cache_control on content blocks. The cached prefix follows the order tools → system → messages. The response usage reports cache_read_input_tokens and cache_creation_input_tokens separately from input_tokens. Cache entries expire after a short TTL, which resets each time the entry is used.
• ▹OpenAI: caching turns on automatically once a prompt passes a minimum length. Look for usage.prompt_tokens_details.cached_tokens in the response.
• ▹Other providers and self-hosted stacks (vLLM, SGLang prefix caching) work the same way: exact prefix match, and the longest match wins. Check your provider's docs for the minimum cacheable size and the TTL.

The hidden cache-busters we found

After finding the timestamp, we audited the rest of the harness. None of these looked like bugs on their own:

• ▹Current date or time in the system prompt. Even at day granularity, it breaks the cache for every live session at midnight. At second granularity it breaks it on every turn.
• ▹Timestamps and request IDs in tool output. A tool returning {"fetched_at": ..., "request_id": ...} is harmless on the turn it's added, because it's at the end of the prompt. It becomes a problem if you re-render or re-serialise earlier tool results differently on the next turn.
• ▹File lists in nondeterministic order. os.listdir, glob, or a set converted to a list can return the same files in a different order. The content is the same, the bytes aren't, and the cache misses.
• ▹Tool definitions in a different order between calls. If you build the tools array from a dict, a plugin registry or a feature-flag filter, the order can change. Tools sit at the very start of the prefix, so a change there breaks the cache for everything after it.
• ▹JSON serialisation drift. Unsorted keys, a different float format or different whitespace between the code that wrote a message and the code that replays it.
• ▹'Helpful' in-place rewrites. If you truncate or summarise an old tool result in the middle of the context to 'save tokens', everything after it gets processed from scratch again.

The layout rule

It works like memory layout in a hot loop: keep the stable data together at the front and put the parts that change at the end. Order the prompt by how often each part changes, least often first.

• ▹Layer 1 should be byte-identical for every request in a session, and ideally across sessions too.
• ▹Layer 2 is loaded once per session and then frozen. If the repo changes, add a new message to the history instead of editing layer 2.
• ▹Layer 3 only grows. Old turns are never edited, re-serialised or reordered.
• ▹Layer 4 holds whatever is true right now: the date, the plan checklist, snippets retrieved for this step, and the new message.

How-to: instrument it yourself

You can't spot this problem by feel, so measure it. Log three numbers per turn: TTFT measured client-side on a streaming call, total input tokens, and cached input tokens. You can get all three from the stream and the usage block.

import time, json, anthropic

client = anthropic.Anthropic()

def run_turn(session_id, turn, tools, system, messages, log):
    t0 = time.perf_counter()
    ttft = None
    with client.messages.stream(
        model="claude-sonnet-5-5",
        max_tokens=2048,
        tools=tools,
        system=system,
        messages=messages,
    ) as stream:
        for event in stream:
            if ttft is None and event.type == "content_block_delta":
                ttft = time.perf_counter() - t0
        final = stream.get_final_message()

    u = final.usage
    cached = u.cache_read_input_tokens or 0
    written = u.cache_creation_input_tokens or 0
    total_in = u.input_tokens + cached + written
    log.write(json.dumps({
        "session": session_id,
        "turn": turn,
        "ttft_s": round(ttft or 0, 3),
        "input_total": total_in,
        "cached": cached,
        "cache_hit_ratio": round(cached / total_in, 3) if total_in else 0,
    }) + "\n")
    return final

Plot cache_hit_ratio and ttft_s against turn number for a few real sessions. The shape of the curve tells you which problem you have:

• ▹Healthy: the hit ratio climbs quickly and stays high, and TTFT stays roughly flat as the session grows.
• ▹Flat near zero: something at the very top of the prompt changes every turn. Usually a timestamp in the system prompt or reordered tools.
• ▹Sawtooth: the hit ratio keeps dropping back. Look for something that changes on a schedule, such as a date rollover, a TTL expiring during long tool calls, or compaction kicking in.
• ▹Steady decline: the cached part stays the same size while the input keeps growing. Usually something rewrites the middle of the history, or the growing history has no breakpoint.
• ▹To find the exact line, diff the serialised request bytes of turn N and turn N+1. The first byte that differs is your cache-buster.

def first_diff(prev_req: dict, next_req: dict) -> str:
    a = json.dumps(prev_req, sort_keys=False)
    b = json.dumps(next_req, sort_keys=False)
    i = next((k for k, (x, y) in enumerate(zip(a, b)) if x != y), min(len(a), len(b)))
    return f"diverges at char {i}: ...{a[max(0,i-80):i+80]!r}"

How-to: fix pass checklist

• ▹Sort everything you list: files, search hits, env vars, dict keys. Use json.dumps(..., sort_keys=True) everywhere you serialise into the prompt.
• ▹Strip timestamps and request IDs from tool output before it goes into the context, or keep only what the model actually needs.
• ▹Move the current date/time out of the system prompt and into the volatile tail. Use day granularity unless the task really needs more.
• ▹Freeze tool definitions at session start: build the list once, in a fixed order, and reuse the same object. Evaluate feature flags at session start, not on every turn.
• ▹Make history append-only. Store the exact bytes you sent and replay them. Don't rebuild old messages from domain objects.
• ▹Put cache breakpoints at the layer boundaries: after tools and system, after project context, and at the end of the history. On Anthropic, that means cache_control on the last block of each layer.
• ▹Re-run the trace on the same scenario and confirm the cached-token count grows along with the input, turn after turn.
• ▹Add a regression test that renders two consecutive turns and asserts that turn N's request is a byte prefix of turn N+1's, up to the volatile tail.

Where layout stops helping

• ▹A growing context still costs you. Cached tokens are cheaper and faster, but not free. The model still attends over them during decode, and very long contexts still hurt quality (Day 'context windows').
• ▹Compaction and summarisation reset the cache. Rewriting history into a summary changes the prefix from that point on. Do it rarely and on purpose, at a natural boundary like the end of a subtask. One cold turn is better than many partial cache misses.
• ▹Cache TTL. If a tool call runs longer than the cache lifetime, the next turn starts cold however well you've laid out the prompt. For long-running tools, look at your provider's longer TTL option.
• ▹Multi-agent handoffs start cold. Each sub-agent has its own prompt, so its own prefix and its own cold first turn. Fan out to five sub-agents and you can pay for five cold prefills, unless they share a byte-identical opening (same tools and system prompt) that's already cached. Tomorrow's capstone covers how to design around this when you put the whole system together.

Takeaway for the series

Prompts are code, and they affect performance like code does. A one-line template change can take an agent with steady latency and make it slower every turn, and no profiler on your own servers will show it. Whoever edits the prompt template is now responsible for latency. Put TTFT per turn and cache hits per turn in the same review you'd give any other hot path. Treat a new {{now}} near the top of a prompt the way you'd treat an allocation inside a tight loop.

Extend your knowledge

• ▹Read the prompt caching sections of Anthropic's and OpenAI's API docs. For the models you use, note the minimum cacheable length, the TTL options and the breakpoint limits.
• ▹Read the vLLM docs on automatic prefix caching and the SGLang paper (which introduced RadixAttention) to see how the same exact-prefix idea works in self-hosted inference.
• ▹Exercise: add the per-turn logger above to your own agent, run a session of 30+ turns, plot hit ratio against turn, and use the byte-diff helper to find your first cache-buster.
• ▹Write a CI test asserting that turn N's request is a byte prefix of turn N+1's (excluding the volatile tail), so a prompt-template edit can't quietly make latency worse.