Model routing in agent loops

Day 20: Route on the Cost of Being Wrong

"Just route easy stuff to the cheap model." Once your agent works and the API bill starts climbing, someone will say this within a week. In an agent loop, that advice can push the bill up. Today you'll learn a routing rule that holds up in production, and why the easiest-looking tasks are sometimes the ones you can least afford to get wrong.

The router that raised our bill by 30%

We shipped a router that sent short, simple-looking prompts to a cheap model. The month we turned it on, our bill went up about 30%. Nearly all of that increase came from tickets like this one.

• ▹The ticket: "Bump the payment service HTTP timeout from 5s to 10s." It was one line of YAML. The router saw a short prompt with no hard keywords and sent it to the cheap tier.
• ▹The quiet mistake: the cheap model edited the right key in the wrong overlay. It changed the base config, not the prod override, so prod still read 5s. The YAML parsed, the schema check passed and nothing went red.
• ▹The retry: after deploy, the check still showed 5s in prod. The agent retried on the cheap model, and it made the same edit again in a slightly different form. Two attempts and two commits got us nowhere.
• ▹The escalation: after the second failure, the loop escalated to the frontier model. It had to re-read the repo, work out what the earlier attempts did, revert them and then make the right edit. Its context was now full of failed attempts, so this call cost more than a clean frontier call would have.
• ▹The aftermath: two invoices, one per provider, for a one-line change. The on-call engineer got paged, found three agent commits on the same key, and spent 40 minutes working out which one was real.
• ▹The lesson: the task was easy, but being wrong about it was expensive, and no cheap check could catch the mistake. Our router was measuring the wrong thing.

Where this fits in the course

On Day 19 we made the agent loop observable. Every step now leaves a trace with model, tokens, cost, outcome and retries. Today we use those traces to pick the model for each step. If you skipped Day 19, go back and do it first, because you can't tune a router you can't measure.

What model routing actually is

Model routing means deciding, before each LLM call, which model will run it. Most setups use one of three shapes, or some mix:

• ▹Static rules. Your code decides from known facts about the step, e.g. 'file reads go to the small model, deploys go to the large one.' This is predictable, cheap and easy to debug.
• ▹Classifier router. A small model or trained classifier reads the prompt and predicts which tier can handle it. RouteLLM works this way. It's flexible but opaque, and it usually judges how hard the prompt looks.
• ▹Cascade/escalation. Try the cheap model, check the output, and escalate to a bigger model if the check fails. The savings depend entirely on how good the check is and how much it costs.

Why 'hard task → big model' breaks in agent loops

In a single chat call, a wrong answer costs you one wrong answer. In an agent loop, a wrong step makes more work. The bad output gets retried, then escalated, then re-run with a bloated context. Downstream steps that built on it have to be redone, and sometimes a human has to debug the result. There's no fixed multiplier for this. A misroute costs more the further the error travels before something catches it, and an error that reaches prod can cost many times what the correct route would have.

Here's the math in illustrative units. Plug in your own numbers from your Day 19 traces. Say a cheap call costs 1 unit and a frontier call costs 10.

Expected cost of cheap-first on a step:
  E = c + (1 - p) * (retries*c + F_escalated + D)

  c           = cheap call cost
  p           = chance cheap is right
  F_escalated = frontier call AFTER failures (bigger context, e.g. 1.5x F)
  D           = downstream rework + human time if the error escapes

Case A: read/summarise a file, wrong output is obvious to the next step
  c=1, p=0.9, retries=1, F_escalated=15, D=0
  E = 1 + 0.1 * (1 + 15 + 0) = 2.6      vs  frontier-only = 10   -> cheap wins

Case B: one-line prod config edit, nothing verifies it before prod
  c=1, p=0.95, retries=2, F_escalated=15, D=300 (incident + on-call time)
  E = 1 + 0.05 * (2 + 15 + 300) = 16.9  vs  frontier-only = 10   -> frontier wins

Look at case B. The cheap model is right 95% of the time, and cheap-first still costs about 1.7x as much. Model quality isn't what drives that number. D is, and D depends on whether a mistake gets caught cheaply or escapes. Prompt length and how hard the task looks tell you nothing about D.

The decision rule: who catches the mistake?

Before you route a step, ask: 'What happens if this step is wrong, and who catches it?' Two properties answer that:

• ▹Reversible or irreversible: can you undo the effect for free? Edits in a sandbox or branch are reversible. Migrations, sent emails, prod config and pushed commits often aren't.
• ▹Verified or unverified: does something automatic check the output before it matters? Tests, type checks, schema validation with real semantics and a diff against expected state all count. 'The model said it looks fine' doesn't.

This rule goes against intuition in both directions. A 500-line refactor looks hard. In a sandbox with a solid test suite, though, most cheap-model mistakes turn into red tests within minutes, so it suits the cheap tier. A one-line prod config change looks trivial, but if it's wrong, your customers are the first to notice. That step goes to frontier, with a human approval gate if you can afford one.

A routing table for a typical coding agent

• ▹Read file → cheap. It changes nothing, and if the summary is bad, the next step that uses the file will correct it.
• ▹Search code/docs → cheap. It's read-only. A bad query just returns poor results, and the agent can search again for pennies.
• ▹Draft a plan → cheap to draft, frontier to approve if the plan drives writes. A draft is throwaway. A plan that gets executed carries the cost of everything built on it.
• ▹Write code in a sandbox with tests → cheap. The work is reversible and verified, and the tests catch mistakes faster and more cheaply than a better model would avoid them.
• ▹Write code with no tests → frontier. Nothing catches a mistake until a human or prod does.
• ▹Run migration → frontier, plus a dry-run and verification. It's hard to reverse, so pay for quality up front.
• ▹Commit/push/deploy → frontier, plus a human gate for prod. It's outward-facing and irreversible, and it's the step the on-call engineer will end up looking at.

Code sketch: a router that routes on consequences

This router never reads the prompt. It decides from facts your agent already knows about the step: the action, whether it writes, whether tests cover it and whether it touches prod. It also caps escalations, so a failing step can't keep bouncing between tiers without anyone noticing.

from dataclasses import dataclass

CHEAP, FRONTIER, HUMAN = "cheap", "frontier", "human"
IRREVERSIBLE = {"migrate", "commit", "deploy"}
MAX_ESCALATIONS = 2

@dataclass
class Step:
    action: str            # read, search, plan, write, migrate, commit, deploy
    is_write: bool
    has_tests: bool
    touches_prod: bool
    escalations: int = 0   # how many times this step already failed and moved up

def route(step: Step) -> str:
    if step.escalations >= MAX_ESCALATIONS:
        return HUMAN       # stop the cascade; page someone with the trace
    if step.touches_prod or step.action in IRREVERSIBLE:
        return FRONTIER    # mistakes here are expensive: pay up front
    if step.is_write and not step.has_tests:
        return FRONTIER    # nobody catches a bad unverified write
    if step.escalations > 0:
        return FRONTIER    # cheap already failed once; don't retry cheap
    return CHEAP           # reads, drafts, writes under tests

Two details matter. First, a step that fails on the cheap model goes straight to frontier instead of retrying on cheap. In our ticket, the second cheap attempt was pure waste. Second, once a step hits the cap, it goes to a human with the trace attached instead of quietly burning money. Tune MAX_ESCALATIONS from your own data, but always set one.

How to know it's working

Every week, pull three numbers per route from your Day 19 traces:

• ▹Cost per completed task, not per call. Adding a cheap tier always makes per-call cost look better, because each call is cheaper. Per-task cost includes the retries, escalations and re-runs, and that's the number on the invoice.
• ▹Escalation rate per route. If a route escalates a lot, either the cheap model isn't suited to that step or the verifier is too weak to trust. Move the step to frontier or fix the check.
• ▹Misroutes caught by tests vs. caught by humans. When tests catch misroutes, the design is working. If the human-caught count goes up, your router is saving tokens by pushing the cost onto people, and engineer hours cost more than tokens.

Common traps for beginners

• ▹Routing on prompt length or keywords. 'Short prompt = easy' is exactly how our prod config edit landed on the cheap model. Length tells you how much text there is. It says nothing about what happens if the answer is wrong.
• ▹Judging the router by classifier accuracy. A router can be 95% accurate and still raise costs if the 5% it gets wrong are prod writes. Measure it by end-to-end cost per completed task and by incidents, not by how often it labels prompts correctly.
• ▹Forgetting that switching models loses context. When you escalate mid-task, the new model doesn't share the old one's KV cache or prompt cache. It has to re-read files and failed attempts, so the escalated call is bigger than a fresh one. Count that cost, and avoid switching models in the middle of a tightly coupled sequence of steps.
• ▹Treating 'the model checked its own work' as verification. A model reviewing its own output doesn't make a step 'verified' in the 2x2. Only count checks that can actually fail: tests, types, dry-runs and real state diffs.

Your Monday exercise

• ▹List every tool call your agent can make. Pull them from your Day 19 traces, not from memory.
• ▹Mark each one reversible or irreversible, and verified or unverified. Be strict about what counts as verified.
• ▹Compare that with your current routing. Look for irreversible or unverified steps on the cheap tier, and for reversible, verified steps on frontier.
• ▹Move one step to the right tier. Track cost per completed task and human-caught misroutes for a week before you change the next one.

On Day 21 we cover the other half of this rule: what to verify and how to do it cheaply. Cheap verification is what lets you move more steps to the cheap tier safely.

Extend your knowledge

• ▹Read the FrugalGPT paper (Chen, Zaharia, Zou, Stanford), the standard reference on LLM cascades. As you read, ask what its scorer catches in an agent loop and what it misses.
• ▹Look at RouteLLM (LMSYS) as an example of a classifier router. Then sort your agent's steps into the ones it could route safely and the ones it shouldn't touch.
• ▹Read Anthropic's 'Building effective agents' post, especially the routing pattern and the sections on human checkpoints and guardrails for agents.
• ▹Use your Day 19 traces to build a per-route dashboard of cost per completed task, escalation rate and human-caught misroutes. Review it weekly before you change any routing rules.