Token budgets per decision

Day 22: Token Budgets per Decision, Not per Task

Most agents that "suddenly get dumb" at step 9 haven't got dumb at all. They're out of attention. By the time they reach the one decision that matters, they've already spent their budget on grep output and file reads that were cheap to make and expensive to carry around. The model isn't the weak part. The allocation is. A per-decision token budget lets you decide where the attention goes before the run starts, instead of finding out from a bad PR afterwards.

The hook: a rename and an architecture call in the same ticket

I see some version of this ticket most weeks: "Rename `PaymentClient` to `BillingGateway` and decide whether the retry logic belongs in the gateway or in the job runner." That's two decisions hiding in one line. The rename is mechanical. If it goes wrong once, the tests catch it. The retry placement decides how the system fails in production for the next year. Here's how one run spent its tokens under a flat per-ticket cap. The numbers are illustrative, but you'll see the same shape in your own traces.

Run #1: flat cap 200k tokens (illustrative trace)

step  type     tokens   
01    explore   18k   ███████
02    explore   22k   █████████
03    explore   25k   ██████████
04    explore   27k   ███████████        <- grep results + 6 full files kept raw
05    explore   29k   ████████████
06    edit      21k   ████████           <- the rename itself, trivial
07    verify    24k   ██████████
08    explore   26k   ██████████         <- "let me check one more file"
09    plan       8k   ███                <- retry placement decision. Cap hit.
                ----
                200k  stopped: budget exhausted

The cap held. The bill was bounded and finance was happy. So what did the cap actually protect? Not the decision that mattered. That decision got 8k tokens, in a context window stuffed with stale grep output. The agent chose "put retries in the gateway" because the gateway file was the last thing it had read. A per-task cap limits total spend. It has nothing to say about where that spend goes.

Where we are in the course

Day 7 covered two things: the context window is finite, and models attend less reliably to material buried in the middle of a long context. On Day 18 you built an agent loop (observe, decide, act, repeat), and every call re-sent the growing transcript. Put those together and you have today's problem. Every raw file the agent keeps in context gets paid for again on every later call, once in tokens and again in diluted attention. Prompt caching lowers the dollar cost of that repetition, but it doesn't make the model look harder at what's buried. So today the context window stops being one bucket and becomes a budget with line items.

The core idea

Here's the whole idea in one sentence: give each type of decision in an agent run its own token share and its own stop rule, and size the share by what the decision is worth, not by how long it takes.

Think about how a working team runs its meetings. Nobody gives the daily standup and the quarterly architecture review the same hour. The standup is short, repetitive and has a hard stop. The architecture review gets the senior people, the prepared docs and enough time to argue properly. When the standup runs long, someone says "take it offline". That's a stop rule.

The theory behind this is Herbert Simon's bounded rationality. A decision-maker with limited time, memory and attention can't optimise everything, so in Simon's account it satisfices: it settles for good enough instead of hunting for the best. In practice that means you spend deliberation where the stakes are. AI research later formalised this as metareasoning, which means deciding how much to think before you think. An LLM agent is a bounded reasoner in the most literal sense, because its budget is counted in tokens. So the rule is: spend in proportion to what the decision is worth, not to how many steps it takes. Exploration takes many steps, and each one is worth little. Planning takes a few, and each one is worth a lot.

The four decision types

Nearly every step a coding agent takes is one of four types. They have different shapes, so each needs different context and a different stop rule.

• ▹Explore: find out what exists. It's wide and shallow: listing files, grepping, skimming. It needs file paths, signatures and short excerpts, and almost never needs full files. Stop rule: when the share runs out, write a structured summary (relevant files, key symbols, open questions), drop the raw material and move on to planning. Exploration ends in a summary, not a hoard.
• ▹Plan: decide what to do and why. This one is narrow and deep, and it deserves the best context you have: the exploration summary, the ticket, any relevant design docs or ADRs, and the two or three files that really matter, loaded fresh. Stop rule: produce a written plan with an explicit decision and its rationale. If the share runs out before there's a plan, hand back to a human.
• ▹Edit: apply the plan. Small, local, cheap. It needs the plan step it's executing and the target file region, and nothing more. Stop rule: one edit per planned change. If the edits keep going past the plan, stop. The plan was wrong.
• ▹Verify: check the work. It's structured and repeatable: run tests, linters and type checks, then diff against the plan. It needs the plan, the diff and the tool output, with failures cut down to the relevant lines. Stop rule: a fixed number of fix-and-retry cycles. After that, hand back to a human with the failing output attached.

Type     Starting share   Context to load                          Stop rule when share runs out
-------  --------------   ---------------------------------------  ---------------------------------------
explore  ~15%             paths, signatures, short excerpts        summarise, drop raw, advance to plan
plan     ~40%             summary + ticket + docs + key files      hand back: "I can't decide this"
edit     ~20%             one plan step + target file region       hand back: "edits exceed the plan"
verify   ~25%             plan + diff + truncated tool output      hand back with failing output

Shares are a starting point for refactor-style tickets, not a law.
Your own data (see below) should replace them.

Worked example: the same ticket, run twice

Same rename-plus-retry ticket, same model. Run A gets a flat 200k cap. Run B gets the same 200k, split 15/40/20/25: 30k for explore, 80k for plan, 40k for edit and 50k for verify. The figures are illustrative. Here's what changes at each step.

• ▹Exploration. Run A opens six full files and keeps every one in context, so each later call re-sends all of them. Run B hits its 30k explore share after about four steps, and the stop rule fires. It writes a summary of roughly 2k tokens: the call sites of PaymentClient, the two places retries currently happen, and one open question about idempotency keys. Then the raw files leave the context.
• ▹Planning. Run A starts planning at about 190k used, with the retry question buried under 150k of stale text. Run B starts with a clean context: the 2k summary, the ticket, the retry section of the job runner and the gateway's error handling (both loaded fresh), and an ADR on idempotency. It has 80k to think with and uses only a fraction of it. Its conclusion is that retries belong in the job runner, because the gateway can't know whether a charge is safe to repeat. A senior engineer would make the same call.
• ▹Editing. Both runs do the rename for roughly the same cost. The difference is that in run B each edit sees only its plan step and target region, so the input per call stays small.
• ▹Verification. Run A never really gets here. Run B runs the test suite and one test fails on a mocked import path. The agent fixes it in one cycle and finishes with budget left over.
• ▹Result. Both runs had the same total budget. Run B spent less overall and put most of what it did spend into the one decision that mattered.

The implementation: about 30 lines around your loop

You don't need a framework for this. Take the Day 18 loop and make three changes: tag each step with its decision type, track tokens per tag, and check the share before running each step. The agent object is whatever you already have. All it has to do is report a step's type and a rough token estimate before it runs.

from dataclasses import dataclass, field

SHARES = {"explore": 0.15, "plan": 0.40, "edit": 0.20, "verify": 0.25}

@dataclass
class DecisionBudget:
    total: int
    shares: dict = field(default_factory=lambda: dict(SHARES))
    spent: dict = field(default_factory=lambda: {k: 0 for k in SHARES})

    def remaining(self, kind: str) -> int:
        return int(self.total * self.shares[kind]) - self.spent[kind]

    def charge(self, kind: str, usage) -> None:
        # Anthropic reports cached prompt tokens separately from input_tokens.
        cached = (getattr(usage, "cache_read_input_tokens", 0) or 0) + \
                 (getattr(usage, "cache_creation_input_tokens", 0) or 0)
        self.spent[kind] += usage.input_tokens + cached + usage.output_tokens


def run(agent, ticket, budget: DecisionBudget, ask_human):
    state = agent.start(ticket)
    while not state.done:
        step = agent.next_step(state)              # step.kind in SHARES
        if budget.remaining(step.kind) < step.estimate:
            if step.kind == "explore":
                # Stop rule for explore: compress and continue, never hoard.
                state = agent.summarise_and_advance(state, to="plan")
                continue
            # Every other type: stop at a natural checkpoint.
            return ask_human(state, reason=f"{step.kind} share exhausted",
                             spend=dict(budget.spent))
        result = agent.execute(step, state)
        budget.charge(step.kind, result.usage)
        state = agent.update(state, result)
    return state

A few details matter here. `result.usage` is the per-call usage object your provider already returns. Field names differ: Anthropic's Messages API and OpenAI's Responses API use `input_tokens` and `output_tokens`, while OpenAI's Chat Completions API uses `prompt_tokens` and `completion_tokens`, so normalise them to one shape. Input tokens include the whole re-sent transcript. That means a hoarded file gets charged to the type of every later step, which is exactly the pressure you want. There's one catch. With prompt caching on, Anthropic reports cached tokens in separate `cache_read_input_tokens` and `cache_creation_input_tokens` fields, not in `input_tokens`. Add them back, as the code does, or hoarded context will look free. `summarise_and_advance` is a single cheap model call that turns raw exploration into a structured note and resets the context to that note. Its cost is charged to explore, so leave a small reserve in the explore share to pay for it.

Why hand back instead of raising the cap? Because running out of a share tells you something. If plan runs out, the agent couldn't make the decision even with good context, and that usually means a human should make it anyway. If verify runs out, something is failing that the agent can't fix in a few cycles. Either way, the handback lands at a natural checkpoint with a summary, a plan or a failing test attached, so you get a review point for free. Compare that with a run that keeps going, burns another 100k, and opens a confident PR built on a bad decision.

Picking your first numbers without guessing

• ▹Log before you limit. Reuse your Day 20 tracing setup and add one attribute to each LLM span, for example `decision.type = explore|plan|edit|verify`. Then run your agent as normal for a week, with no new limits.
• ▹Find where the tokens went. Group by ticket type and decision type, and check both the median and the worst runs. In the traces I've looked at, exploration is usually much bigger than the team expected, and planning much smaller.
• ▹Compare spend with outcomes. Pull the runs you rejected or reverted and find where each one went wrong. If the failures go back to plans made with starved context, planning needs a bigger share.
• ▹Set shares from the data plus your judgement. Start from the distribution you observed, move tokens from low-value types to high-value ones, and write down why you did it. Check again after another week.
• ▹Size per ticket class, not globally. A dependency bump and a cross-service refactor shouldn't use the same split. Two or three profiles are enough to start.

Failure modes to watch

• ▹Shares so tight the agent thrashes. You'll see it summarise, re-explore and summarise again, or hand back on nearly every run. Add a minimum floor per type and keep an eye on the handback rate. A handful a week is useful. Several a day means your shares are wrong.
• ▹Misclassified steps. If the agent labels a deep reasoning step as "explore", that step gets cut off early. If it labels a file read as "plan", the plan share goes on browsing. Where you can, classify from the action itself (file reads and greps are explore, writes are edit, test runs are verify) instead of trusting the model's label.
• ▹Plan budgets that hide a vague spec. When plan keeps running out on one kind of ticket, the agent usually isn't short of tokens. It's trying to resolve ambiguity that only a human can resolve. Fix the ticket, not the budget. A bigger plan share on a vague ticket just buys a longer, more confident guess.
• ▹Summaries that drop the important detail. If compression loses the one constraint the plan needed, the plan goes wrong and nothing flags it. Give the summary a fixed schema (files, symbols, constraints, open questions) and check a sample by hand.
• ▹Treating shares as cost control only. What you're after is decision quality. If total spend drops but rejected PRs go up, the split is wrong.

Exercise for today

• ▹Pick one agent task you already run, such as a refactor, a bug fix or a docs update.
• ▹Tag every step as explore, plan, edit or verify, ideally based on the tool being called.
• ▹Record input and output tokens per step from the API usage object, and include any cached-token fields.
• ▹Build a per-decision spend chart: one stacked bar per run, one colour per type. A terminal bar chart like the hook trace is fine.
• ▹Write two sentences. Where did the tokens go, and was that where the important decision was?
• ▹Bonus: add one stop-and-ask-a-human rule, for example "if plan exceeds its share, hand back the exploration summary and the open questions". Run five tasks and count how often it fires.

Bridge to Day 23

Once you know what each decision costs and what it's worth, the next question follows naturally: why is the same model making all of them? Exploration and editing are cheap, repetitive and easy to check. Planning is rare and high-stakes. Tomorrow is model routing, which means sending each decision type to the model that suits it: a small, fast model for exploration and your strongest model for the plan.

Extend your knowledge

• ▹Herbert A. Simon, "A Behavioral Model of Rational Choice" (1955): where bounded rationality was first framed.
• ▹Stuart Russell and Eric Wefald, "Do the Right Thing: Studies in Limited Rationality" (1991): metareasoning, i.e. deciding how much computation a decision deserves.
• ▹Anthropic's engineering posts "Building effective agents" and "Effective context engineering for AI agents": practical patterns for keeping agent context lean and deliberate.
• ▹Your own traces: run this week's exercise again on a second ticket class and compare the two spend charts. The difference between them is your first routing signal for Day 23.