Token cost in multi-agent fleets

Day 28: Where the Tokens Actually Go in a Multi-Agent Fleet

Your LLM bill can keep climbing even when the problems your agents solve are no harder than last month's. Once an orchestrator runs a handful of sub-agents, the bill stops following how hard the work is and starts following how information moves between agents. If you don't measure that flow, you'll fix the wrong thing. The usual wrong thing is switching to a cheaper model.

The lesson in one line: in a multi-agent fleet, the expensive token is the one spent re-reading context, not the one spent reasoning. The fix is in the architecture. Changing the model won't do it.

1. The week we tagged every token by purpose

Our invoice showed how much we spent but not what we spent it on. So for one week we wrapped every LLM call in the fleet (one orchestrator, a dozen sub-agents, real repos) and tagged each segment of each prompt with its purpose. Nothing clever was needed. The tagger already knew which part of the prompt it was building, so it recorded the token count of each part before sending.

• ▹System and conventions: the system prompt, coding standards and tool definitions
• ▹Spec: the task or ticket description and acceptance criteria
• ▹Repo files: source files pulled into context
• ▹Prior transcript: earlier conversation passed along in a handoff
• ▹Tool output: test runs, grep results, linter output
• ▹Reasoning and output: what the model actually generated

The first four buckets are re-ingested context. The model has to read them, but in most cases another agent in the fleet has already read the same material. Only the last bucket is the work we thought we were paying for.

The result was lopsided. Re-ingested context took the large majority of tokens. The biggest single bucket was repo files that more than one agent read for the same task. Reasoning and output were a thin slice. Put plainly, model choice wasn't driving cost. Volume was, and most of that volume was duplicated reading.

Run this on your own fleet before you trust any number in this lesson, including mine. The proportions shift with workload. Which way the split leans usually doesn't.

2. The reflex we almost followed: smaller sub-agents

The obvious move was to keep a strong model on the orchestrator and push the sub-agents down to a smaller, cheaper one. We nearly did it. Here's why it's the wrong fix.

• ▹It lowers the price per token and leaves the token count alone. If most of your spend is re-reading, you're now paying less per unit for the same waste.
• ▹Weaker sub-agents get confused more often, and confused agents read more. They re-open files 'to be sure', ask for the spec again, or run the same grep several times. Part of the price saving comes straight back as volume.
• ▹Their mistakes go back to the orchestrator as rework. Rework means another handoff, and every handoff carries another full-context payload.
• ▹It hides the real problem. The bill dips a little, and nobody goes back to look at the information flow that's causing it.

Smaller models do have a place: narrow, well-defined sub-tasks with small inputs. But choosing one is a capability decision. Make it after you've fixed the information flow, not instead of fixing it.

3. The mechanism: every handoff was a full-context handoff

This section draws on two earlier days of the course. From the handoff day: a handoff decides what the next agent knows. From the shared-memory day: agents can share beliefs instead of each building their own. Our fleet did neither well. Every handoff carried everything (system prompt, spec, relevant files, the conversation so far). Nothing was shared, so every agent rebuilt its own picture of the task from scratch.

That makes cost grow multiplicatively. Useful information grows slowly, because a task only has so many facts. Tokens grow with agents × handoffs × context size.

Illustrative arithmetic (not our real numbers):

shared context C   = spec + conventions + relevant files + transcript so far
agents N           = 12
handoffs per agent = M  (initial brief + replies + rework rounds)

input tokens  ≈ N × M × C          # and C itself grows as the transcript grows
new info      ≈ roughly constant   # the task has a fixed number of facts

Double the agents  -> roughly double the bill
Add a rework round -> another N × C
Transcript in C    -> each round is more expensive than the last

Every sub-agent drinks from the same firehose, and the transcripts flowing back make it bigger for the next round.

4. Audit your own fleet in an afternoon

You don't need a new platform for this. You need logs that record, for every call, which agent made it, its input and output token counts, its cached-token count (most providers return this in the usage field), and which files went into the prompt. Then make three cuts.

• ▹Cut 1: input vs. output tokens per agent. A coding sub-agent that reads far more than it writes can be perfectly healthy. If an agent's ratio keeps climbing over a task, that agent is re-reading. What we saw: every sub-agent was input-heavy, and the ratio grew with each handoff round as transcripts piled up.
• ▹Cut 2: duplicate-read rate. Hash each file's content as it goes into a prompt. Per task, count the file hashes that more than one agent read. What we saw: the core files (the module under change, its types, the main config) were read by nearly every sub-agent on the task, often more than once each.
• ▹Cut 3: cache hit rate on the prompt prefix. Divide cached input tokens by total input tokens, per agent. What we saw: close to zero, even though every agent got the same system prompt and spec. The cause turned out to be prompt ordering (see Fix 1).

# Assumes one log row per LLM call:
# {task_id, agent, input_tokens, output_tokens, cached_input_tokens, files: [{path, sha}]}
from collections import defaultdict

def audit(calls):
    per_agent = defaultdict(lambda: {"in": 0, "out": 0, "cached": 0})
    readers = defaultdict(set)          # (task_id, sha) -> agents that read it

    for c in calls:
        a = per_agent[c["agent"]]
        a["in"] += c["input_tokens"]
        a["out"] += c["output_tokens"]
        a["cached"] += c.get("cached_input_tokens", 0)
        for f in c["files"]:
            readers[(c["task_id"], f["sha"])].add(c["agent"])

    for name, a in per_agent.items():
        ratio = a["in"] / max(a["out"], 1)
        hit = a["cached"] / max(a["in"], 1)
        print(f"{name:20} in/out={ratio:6.1f}  cache_hit={hit:5.1%}")

    dup = sum(1 for agents in readers.values() if len(agents) > 1)
    print(f"duplicate-read rate: {dup / max(len(readers), 1):.1%} of (task, file) pairs")

5. Fix 1: cache-friendly prefixes

The major providers (Anthropic, OpenAI, Google) offer prompt caching. If the start of your prompt exactly matches a recently processed prompt, those tokens are served from cache, which is cheaper and faster. The word that matters is prefix. The match starts at the first token and stops at the first difference. (In practice providers match in fixed-size chunks or at cache breakpoints, not token by token.) Everything after that difference is processed at full price.

So order your prompt from most stable to most volatile, and use the same order for every agent in the fleet:

[1] Tool definitions            (identical for all agents, stable ordering)
[2] Fleet system prompt         (shared rules, no per-agent text here)
[3] Conventions + repo map      (regenerated only when the repo changes)
[4] Task spec                   (stable for the life of the task)
    ---- cache breakpoint ----
[5] Agent role                  ("You are the test-writer sub-agent")
[6] Handoff payload             (goal, decisions, diffs, open questions)
[7] Current turn / tool output  (changes every call)

The ordering mistakes below break caching without any error, so you have to check for each one yourself:

• ▹A timestamp, request ID or task ID in the system prompt. One changing token near the top invalidates everything after it.
• ▹The agent's persona placed first. Putting 'You are the reviewer' ahead of the shared context gives each agent a different prefix, so no two agents can share a cache.
• ▹Non-deterministic serialisation: tool lists built from a set or dict with unstable ordering, or JSON with unsorted keys. The content is the same but the bytes differ, so there's no cache hit.
• ▹Regenerating the repo map on every call with small differences (file order, whitespace) when the repo hasn't changed.
• ▹A prefix shorter than the provider's minimum cacheable length, which means it never gets cached. Check your provider's docs for the current threshold.
• ▹Cache TTL expiry. Default cache lifetimes are often short, in the range of minutes for automatic or default caching. If sub-agents start long after the orchestrator warmed the cache, the cache has already expired. Check your provider's TTL options.
• ▹Mixing models. Caches are per model, so a sub-agent on a different model can't reuse the orchestrator's cache.
• ▹On providers with explicit cache breakpoints (for example Anthropic's cache_control): forgetting to set one, or placing it after volatile content.

6. Fix 2: the context ledger

Caching makes repeated reading cheaper. The ledger makes it unnecessary. It's one shared record per task of what has been read, what was concluded from it and what was decided, with each entry keyed by the file's content hash. In MAS terms this is a shared belief base. Agents check a common store instead of building private beliefs from raw percepts, and they only go back to the raw source when the store can't answer.

{
  "task_id": "T-412",
  "files": {
    "src/billing/invoice.ts": {
      "sha": "9f2c1e…",
      "read_by": "explorer",
      "summary": "Builds invoice lines; tax applied in applyTax(); currency is minor units (int).",
      "key_symbols": ["buildInvoice", "applyTax", "InvoiceLine"]
    }
  },
  "decisions": [
    {"id": "D3", "by": "orchestrator", "text": "Keep minor-unit integers; no floats in billing."}
  ],
  "open_questions": ["Does the refund path reuse applyTax?"]
}

def get_context(ledger, path, need):
    """Ask the ledger before reading. need = 'summary' | 'full'."""
    current_sha = sha_of(path)                 # cheap: hash on disk, no LLM call
    entry = ledger.files.get(path)

    if entry and entry.sha == current_sha and need == "summary":
        return entry.summary                   # hit: no re-read

    content = read_file(path)                  # miss, stale, or full text needed
    if not entry or entry.sha != current_sha:
        ledger.files[path] = summarise_and_record(path, content, current_sha)
    return content

The design depends on two rules. First, an agent can always ask for the full file. The ledger is a default, not a wall, and an agent about to edit a file should read the real thing. Second, every summary is bound to a hash, so you can always tell whether it still describes the current code.

7. Fix 3: diffs, not transcripts

A handoff should carry what the next agent needs in order to act, not everything that happened. That means the goal, the decisions made, the changed files as diffs, and the open questions. The conversation stays in the logs.

BEFORE: transcript handoff                AFTER: decision handoff
--------------------------------------    --------------------------------------
system prompt (again)                     goal: "Add tax to refund path"
spec (again)                              decisions: [D3: minor-unit ints]
full invoice.ts, refund.ts, tax.ts        changed: refund.ts (diff, +18 -4)
38 turns of explorer conversation:        ledger refs: invoice.ts@9f2c1e,
  "let me look at...", tool calls,                     tax.ts@a71b03
  grep output, dead ends, retries         open: "Refund path reuses applyTax?"
final message: "done, see above"          acceptance: tests in refund.spec.ts pass

size: grows with every turn               size: roughly constant per handoff

The transcript version feels safer because nothing gets lost. But the receiving agent then has to find the three facts it needs among all the dead ends, and sometimes it acts on a dead end as if it were a decision. The decision version is smaller and clearer.

8. The after picture, honestly

We ran the same tasks on the same models and got a different breakdown. Re-ingested context shrank sharply. The cache hit rate on the shared prefix went from close to zero to being the main way stable context got served. Reasoning plus output became a much larger share of a smaller bill. The fleet also got faster, because cached prefixes are quicker to process.

Some things got worse, and you should expect them too:

• ▹Ledger staleness. A sub-agent acted on a summary of a file that another agent had just edited. The summary was accurate, but only for the old version. Guard: every ledger read re-hashes the file on disk, and a mismatch forces a re-read. Edits write the new hash to the ledger before the handoff goes out.
• ▹Lossy summaries. A summary left out an edge case that the next agent needed. Guard: summaries list key symbols, and any agent about to modify a file must read it in full.
• ▹Over-compressed handoffs. When we dropped the transcript, we sometimes dropped the reason behind a decision as well. Guard: every decision carries a one-line rationale, and an agent can still fetch the full transcript by ID if it explicitly asks.
• ▹More moving parts. The ledger is now infrastructure. It needs tests, concurrency control for parallel writers, and cleanup when a task ends.

9. When this doesn't apply

• ▹Single-agent setups: there are no handoffs to fix. Prefix caching still helps, but the ledger and diff handoffs mostly don't.
• ▹Tiny repos and short tasks: the context is small enough that duplicate reads cost little, and running the ledger costs more than it saves.
• ▹Reasoning-heavy work: hard algorithm design, long proofs, deep debugging on a small surface. Output and thinking dominate here, so model choice really is the lever.

Quick rule: run Cut 1 and Cut 2. If input tokens heavily outweigh output across agents and your duplicate-read rate is high, you have an information-flow problem, so fix the architecture. If output and reasoning dominate and duplicate reads are rare, you have a reasoning problem, so tune the model and the prompts.

10. Bridge to Day 29

Today's point: cost follows information flow, not model price. Tomorrow we look at the other bill you pay, which is time. Adding agents often makes a fleet slower, because the orchestrator waits on the slowest sub-agent and that tail sets the pace. Day 29 covers latency in multi-agent systems.

Extend your knowledge

• ▹Read your provider's prompt caching docs (Anthropic's prompt caching guide, OpenAI's prompt caching guide, Google's context caching docs). Note the minimum cacheable length, the TTL, and how cached tokens appear in the usage response.
• ▹Read Anthropic's engineering post on how they built their multi-agent research system. It covers why multi-agent setups use far more tokens than single-agent chat, and why sub-agents should return condensed results to the lead agent.
• ▹Add purpose tags to your own prompt builder this week, then run the three audit cuts on one real task before you change anything.
• ▹Revisit the BDI (belief-desire-intention) model from the agent literature (e.g., Wooldridge's 'An Introduction to MultiAgent Systems') and map your context ledger onto shared beliefs.