Open vs closed models in agents

Day 28: Open vs Closed Models in Agent Systems, Pick Per Seat

"Just use open source, it's cheaper." "Just use the best API." You've heard both, and both are right. They're just about different parts of your system. Once you build agents, stop asking which model you use. Ask which seat each model sits in.

The hook: read your token bill by role

Our first internal research agent ran one strong frontier model for everything. It planned the task. It also summarised, extracted, reformatted and classified. Then we tagged every call by role and split the bill, and the answer was obvious. Most tokens went to grunt work: reading retrieved pages, pulling out fields, turning text into JSON. The planning calls were the whole reason we paid for a frontier model, and they were a small share of the volume.

We were paying frontier prices for work a much smaller model could handle. That raised the question for this lesson: open or closed, and for which seat?

If you keep one habit from today, make it this one. Log a role tag with every model call. If your bill only shows one total, you can't decide anything per seat.

60-second definitions

• ▹Closed model: you rent it through an API (Claude, GPT, Gemini). Tokens go in, tokens come out. You can't see the weights or run the model on your own hardware, and the provider decides when versions change or get retired. What you get back: the frontier, which is usually the strongest model available, and no infrastructure to run.
• ▹Open-weight model: the trained weights are published. You can download them and run the model yourself (vLLM, SGLang, Ollama, llama.cpp) or through any host you like. Llama, Qwen, Mistral, DeepSeek and gpt-oss are all examples. You control the exact version, where your data goes and how it's served.
• ▹What 'open weights' does NOT mean: it doesn't automatically mean open training data, open training code or an OSI-approved open source licence. Some licences limit commercial use or scale. Read the licence before anything goes to production.

Callback: agent systems have roles

Earlier in the series we built the basic agent loop: the model thinks, calls a tool, reads the result and goes again. When a task outgrows one loop, you split it into roles. A planner (or orchestrator) breaks the goal into steps and hands them out. Workers do narrow jobs like searching, extracting, summarising or writing a test. A checker, if you have one, validates worker output before it goes back to the planner. Every role is its own set of model calls, so every role can use a different model.

The core idea: judgment vs volume

The planner does judgment work. It decides how to split the task, chooses the next move, decides whether a result is good enough and recovers when a step fails. When the planner gets something wrong, the whole structure is wrong. Split the task badly and every worker does the wrong job, and does it well. Read a failed step as a success and everything after it builds on bad data. Planner errors spread. They often reach the end of the run without anyone noticing, because each worker's output looks fine on its own.

Workers do volume work. Each one gets a narrow, well-specified job: extract these fields, summarise this page in three bullets, label this ticket. They run many times per task with the same instructions every time, and the output usually has a shape you can check. A worker mistake usually stays local, and a schema check, a test run or a retry catches it before it reaches anything else.

Together, those two facts give you one rule. The token bill sits with the workers. The failures that really hurt start with the planner. Spend on judgment, save on volume.

The seat-assignment test

Ask three questions about each role:

• ▹1. Blast radius: when this role gets it wrong, does the error spread to later steps, or does something catch it?
• ▹2. Volume: how many calls does this role make per task? One? Five? Hundreds?
• ▹3. Checkability: can I verify its output automatically, for correctness and not just format? Schema validation, unit tests, exact-match against a source and rule checks all count. 'It looks fine' doesn't.

How to read the answers:

• ▹Errors spread, few calls, hard to check → closed frontier model. That's the planner seat. There are so few calls that the per-token price barely matters.
• ▹Errors get caught, lots of calls, automatically checkable → an open-weight model you host. These are your worker seats, and this is where the money is.
• ▹Mixed answers (lots of calls but hard to check, or few calls with a big blast radius) → start on closed and measure. Only move the role once you can show the open model holds up. More on that below.

# seat-map.yaml : one entry per role, reviewed whenever a model changes
planner:
  model: closed-frontier        # e.g. a top Claude / GPT / Gemini tier
  why: errors spread, ~1-5 calls per task, output is a plan (hard to auto-check)

worker_extract:
  model: open-weight-mid         # e.g. a Qwen / Llama instruct model on vLLM
  pinned: model-name@revision-hash, quant=bf16, temperature=0
  why: high volume, JSON schema + source-match check catches errors

worker_summarise:
  model: closed-frontier         # moved BACK, see 'how it went'
  why: format check passed, faithfulness was not being checked

checker:
  model: none (code)             # deterministic checks where possible
  why: cheapest, most predictable judge is code

How it went for us

What got cheaper: the extraction and classification workers. They made most of the calls, their output fit a strict JSON schema, and we could check extracted values against the source text in code. Moving them to a self-hosted open-weight model cut the biggest part of the bill. On our tasks the open model kept most of the quality, and the checker caught the rest with retries. That's our workload, not a general benchmark. Yours will behave differently.

What stayed put: the planner. We tried a strong open model there for a short time. Each plan looked reasonable in isolation, but recovery after a failed step was noticeably worse, and those failures only appeared at the end of a run. Planner calls were a small share of tokens anyway, so switching would have saved very little. It went back to a frontier model and has stayed there.

The one we had to move back: the summarising worker. On paper it passed all three questions: lots of calls, 'checkable' output. But the checker only verified format: three bullets, under a length limit, valid JSON. Nothing checked whether the bullets were faithful to the source. The open model sometimes merged two claims or dropped a qualifier, and the planner trusted those summaries. So the errors spread, even though the role was called a 'worker'. We moved it back to a frontier model until we have a real faithfulness check. The lesson: a role only counts as checkable if your checks catch the errors that matter. Checking the shape isn't enough.

What you give up

Ops cost. Self-hosting means GPUs, an inference server, autoscaling, monitoring, cold starts and someone on call when it falls over. At low volume, a cheap closed API tier or a managed open-model host often costs less than running your own. Self-hosting pays off at sustained high volume, or when you have strict rules about where data can go. Compare the full cost, including engineer time, and not just the price per token.

Pinning and settings. Open models only give you stable versions if you pin them. Fix the exact weights revision, the quantisation, the chat template, the inference server version, temperature and max tokens. Change any one of them and outputs can change. Even fully pinned, batched GPU inference can drift slightly between runs at temperature 0, so keep your checks in place. A big reason to use open weights is that nobody can swap the model out from under you. If you don't pin, you lose that.

Handoffs between model families. Put a planner from one family in front of workers from another, and the boundary becomes a new place for things to break. Tool-call formats differ. So does how strictly each family follows JSON instructions and how it reads terse instructions. A planner can write a step its own family would understand perfectly and an open worker would misread. Make the contract at the boundary explicit with schemas, examples and strict validation. Don't rely on the two models sharing assumptions.

Try it today

Pick an agent workflow you use or can picture: a research assistant, a code reviewer, a support triage bot. Then:

• ▹List every role, including the ones hidden today because one model does everything (for example, 'decide which files to review' is a different role from 'comment on one diff hunk').
• ▹Estimate calls per task for each role. A rough order of magnitude is fine.
• ▹Run the three-question test on each role. For question 3, write down the actual check ('JSON schema + value appears in source'), not just 'yes'.
• ▹Fill in a seat map with one line per role: role → open or closed → why.
• ▹Flag any role whose check verifies format but not correctness. Those roles stay closed until the check gets better.

Example: AI code reviewer

role              calls/PR  errors spread?        auto-check?               seat
----------------  --------  --------------------  ------------------------  ------
review planner    1-2       yes (wrong focus)     no                        closed
hunk commenter    20-100    no, human reads it    partial (lint, dedupe)    open*
test-gap finder   5-10      no                    yes (run coverage)        open
security triage   1-5       yes (missed vuln)     weak                      closed

* start closed, move once you have an eval set that shows parity

Bridge to Day 29

Once each seat can take a different model, you'll keep swapping them: a new open release, a cheaper closed tier, a new quantisation. Every swap raises the same question: better or worse? If you can't measure that, you're going on gut feel, and our summariser shows how a swap can look fine and still be wrong. Tomorrow is evals: how to build a small, honest test set for each role so a model swap becomes a measured decision.

Extend your knowledge

• ▹Read Anthropic's 'Building effective agents' post, especially the orchestrator-workers pattern, and map its roles onto the seat-assignment test.
• ▹Serve a small open-weight instruct model locally with Ollama or vLLM. Run one narrow worker task (field extraction) at temperature 0 and compare it with a frontier API on 20 examples.
• ▹Add a role tag to every model call in a project you own and break down token spend by role for a week. That breakdown is your first seat map.
• ▹Pick one open-weight model you're considering (Llama, Qwen, Mistral or gpt-oss), read its licence and write down exactly what it allows and what it restricts.