Cost per accepted change
Day 23: Cost per accepted change, the metric your API invoice doesn't show
Your API invoice counts tokens. Your team pays for shipped changes. Those are two different numbers. If you shrink the invoice by making the agent worse, the cost doesn't go away. It moves into your reviewers' hours, and there it's usually much bigger.
1. A 40% saving that cost us more
We once moved most of our coding agent's steps to a cheaper model. The next invoice came in about 40% lower and everyone was pleased. Then someone looked at review load. PRs bounced back to reviewers more often, more got rejected, and a few were merged and later reverted. Once we put the invoice next to reviewer hours, the 'cheap' month turned out to be our most expensive month for agent-assisted work.
Illustrative shape (rounded). The pattern is what matters, not the exact values.
Month A (premium) Month B (cheap) Change
API invoice $310 $186 -40% <- the line on the bill
Reviewer hours ~23 h ~47 h +100% <- the line nobody charts
Merged & not reverted 81 52 -36%
The invoice chart went down. The reviewer-hours chart went up.
Only one of these charts reached the finance review.2. The unit error
Cost per token feels like the right unit because it's what the provider bills you for. It's the wrong denominator, because nobody buys tokens for their own sake. You're paying for changes that ship and stay shipped. A cheap token that produces a PR a human has to rewrite costs more than an expensive token that produces a PR the human approves in five minutes.
cost per accepted change =
API spend
+ review hours x loaded hourly rate
+ rework hours x loaded hourly rate
+ incident/cleanup hours x loaded hourly rate
---------------------------------------------------------
changes merged AND not reverted within N days
N: pick 7 or 14 days and keep it fixed so you can compare periods.
Loaded rate: salary + overhead per hour. Ask finance; they have this number.- ▹Numerator: everything the change cost, human time included. In agent workflows, human time is usually the biggest term.
- ▹Denominator: changes that shipped and survived. A PR that's merged and then reverted on day 3 goes in the numerator (you paid for it, and for the cleanup). It stays out of the denominator.
- ▹'Accepted' needs a definition: merged and not reverted within N days. 'The agent said it was done' doesn't count, and neither does 'tests passed once'.
3. Worked example: same 100 tasks, two setups
The numbers below are illustrative, but the shape is realistic. Setup A runs a cheap model on every step. Setup B uses a premium model for planning and cheap models to carry out the plan. Both run the same 100 tasks, the loaded reviewer rate is $100/h, and each revert costs about 2 hours of cleanup.
A: cheap everywhere B: premium planner + cheap exec
API cost per attempt $0.60 $2.10
Avg attempts per task 3.1 1.48
API spend (100 tasks) $186 $310
Accepted (merged, not reverted) 52 81
Reverted after merge 4 1
Reviewer minutes per PR 28 14
Review + rework hours 46.7 h -> $4,667 23.3 h -> $2,333
Incident cleanup 8 h -> $800 2 h -> $200
---------------------------------------------------------------------------------------
Total cost $5,653 $2,843
Cost per accepted change $108.7 $35.1
API-only cost per accepted change $3.58 $3.83 <- almost the same
Cost per token A wins B loses- ▹Per token, A wins easily. Per accepted change, B costs about a third as much.
- ▹Leave human time out completely and A is still only slightly ahead. Retries and a lower acceptance rate cancel out almost all of the cheaper-token advantage.
- ▹Human time is roughly 90% or more of the total in both columns (about 97% in A, 89% in B). Cutting the smallest term by 40% gets you nothing when the largest term doubles.
4. Where the money goes: the retry tax
Open one agent run log and label each step. Most of the tokens don't go to the first real attempt. They go to the agent redoing work it has already done. When we audited our runs, loops were about 40% of API spend. They also caused most of the review pain, because a run that looped usually ended in a messy diff.
One run, annotated (illustrative; tokens = new tokens added by the step, not counting re-sent context):
step action tokens bucket
1 plan 8k first attempt
2 read src/billing/invoice.py 12k first attempt
3 edit invoice.py 6k first attempt
4 run tests -> 2 fail 9k first attempt
5 read src/billing/invoice.py 12k RE-READ (unchanged file)
6 read tests/test_invoice.py 7k first attempt
7 edit invoice.py 5k first attempt
8 run tests -> same test fails 9k TEST RE-RUN (same failure)
9 'let me reconsider the approach' 10k RE-PLAN (lost the plan)
10 read src/billing/invoice.py 12k RE-READ
11 read src/billing/tax.py 8k first attempt
12 edit tax.py + invoice.py 9k first attempt
13 run tests -> pass 9k first attempt
first attempt: 73k re-reads: 24k test re-runs: 9k re-plans: 10k
loop share: 43k / 116k = 37% of this runEvery step re-sends the conversation so far, so each extra loop costs more than the one before it. The re-read at step 10 costs more than the same read at step 2 because it carries every token from steps 1 to 9 with it. Prompt caching gives you a discount on that re-sent prefix, but you still pay for it.
Why weaker models loop more
- ▹They lose the plan. Once the context gets long, a weaker model stops tracking what it decided at step 1. So it re-plans, and the new plan often contradicts the old one.
- ▹They hedge. Instead of reasoning about code they've already read, they read it again 'to be sure'. The transcript looks busy, but nothing new is learned.
- ▹They retry blindly. A test fails, they make a small change and re-run it without any idea why it failed. If the same test fails twice in a row, the agent is no longer debugging.
5. Exercise part 1: instrument your traces
Use the observability setup from earlier in the course: Langfuse, LangSmith, OpenTelemetry, or a plain table in Postgres. Add three fields: task_id, attempt and final outcome. Without them you can't link API spend to whether the change actually shipped.
-- One row per task (a ticket, an issue, a PR-sized unit of work)
CREATE TABLE tasks (
task_id TEXT PRIMARY KEY,
final_outcome TEXT CHECK (final_outcome IN ('accepted','rejected','reverted','escalated','open')),
pr_url TEXT,
merged_at TIMESTAMPTZ,
reverted_at TIMESTAMPTZ,
reviewer_minutes INT, -- from PR review tooling or self-reported
rework_minutes INT
);
-- One row per agent attempt at a task
CREATE TABLE runs (
run_id TEXT PRIMARY KEY,
task_id TEXT REFERENCES tasks(task_id),
attempt INT, -- 1, 2, 3 ... restarts from scratch or after human feedback
role TEXT, -- planner | executor | reviewer
model TEXT,
input_tokens INT,
output_tokens INT,
cost_usd NUMERIC(10,4),
started_at TIMESTAMPTZ
);
-- One row per step inside a run (you likely already log this as spans)
CREATE TABLE steps (
run_id TEXT REFERENCES runs(run_id),
seq INT,
kind TEXT, -- plan | tool_call | message
tool TEXT, -- read_file | edit_file | run_tests | ...
target TEXT, -- file path or test id
status TEXT, -- ok | fail
tokens INT
);Set final_outcome from what actually happens to the PR, never from the agent's own report. All you need is a small job that checks merged PRs N days later and changes 'accepted' to 'reverted' when it finds a revert.
6. Exercise part 2: audit
-- A. Retries per accepted change
WITH per_task AS (
SELECT t.task_id, t.final_outcome, MAX(r.attempt) AS attempts, SUM(r.cost_usd) AS api_cost
FROM tasks t JOIN runs r USING (task_id)
GROUP BY t.task_id, t.final_outcome
)
SELECT
SUM(attempts - 1)::float
/ NULLIF(COUNT(*) FILTER (WHERE final_outcome = 'accepted'), 0) AS retries_per_accepted,
SUM(api_cost)
/ NULLIF(COUNT(*) FILTER (WHERE final_outcome = 'accepted'), 0) AS api_cost_per_accepted
FROM per_task;
-- B. Loop pattern 1: same file read more than twice in a task
SELECT r.task_id, s.target, COUNT(*) AS reads, SUM(s.tokens) AS tokens
FROM steps s JOIN runs r USING (run_id)
WHERE s.tool = 'read_file'
GROUP BY r.task_id, s.target
HAVING COUNT(*) > 2
ORDER BY tokens DESC;
-- C. Loop pattern 2: same failing test re-run
SELECT r.task_id, s.target AS test_id, COUNT(*) AS failing_runs, SUM(s.tokens) AS tokens
FROM steps s JOIN runs r USING (run_id)
WHERE s.tool = 'run_tests' AND s.status = 'fail'
GROUP BY r.task_id, s.target
HAVING COUNT(*) >= 2
ORDER BY tokens DESC;
-- D. Loop pattern 3: plan regenerated mid-task
SELECT r.task_id, COUNT(*) AS plans, SUM(s.tokens) AS tokens
FROM steps s JOIN runs r USING (run_id)
WHERE s.kind = 'plan'
GROUP BY r.task_id
HAVING COUNT(*) > 1
ORDER BY tokens DESC;- ▹Add up the tokens from B, C and D and divide by total tokens. That's your loop share. Write it down as your baseline.
- ▹Rank the top 3 patterns by token cost, not by count. One file read 9 times late in a long context can cost more than 50 short re-reads.
- ▹Check which files show up in pattern B. If the same few files keep appearing across many tasks, the agent should be getting that context at the start (fix 1 below).
7. Three fixes that cut the bill by spending more upfront
Fix 1: Better first-attempt context
Most re-reads happen because the agent works out the repo from scratch on every task. Give it what it needs at the start: a short architecture note, the files most likely to be relevant (pulled with the RAG setup from earlier in the course), the conventions, and the exact command for running tests. The first prompt gets bigger, but the loops shrink by far more than that. Repo-level instruction files like CLAUDE.md and AGENTS.md exist for exactly this reason.
Fix 2: A strong model for the planner, cheap models for bounded steps
Losing the thread costs the most during planning, so that's the step where a strong model pays off. Once the plan is explicit (files, edits, expected test results), each execution step has a clear scope and a cheaper model can do it. Write the plan down as an artifact the executors read from, so nobody has to remember it.
Fix 3: Explicit stop conditions and escalation
from collections import Counter
MAX_ATTEMPTS = 3
MAX_SAME_READ = 2
MAX_SAME_TEST_FAIL = 2
class Budget:
def __init__(self):
self.reads = Counter()
self.test_fails = Counter()
self.plans = 0
def check(self, step):
if step.tool == 'read_file':
self.reads[step.target] += 1
if self.reads[step.target] > MAX_SAME_READ:
return 'stop: re-reading unchanged file ' + step.target
if step.tool == 'run_tests' and step.status == 'fail':
self.test_fails[step.target] += 1
if self.test_fails[step.target] >= MAX_SAME_TEST_FAIL:
return 'stop: same test failing again ' + step.target
if step.kind == 'plan':
self.plans += 1
if self.plans > 1:
return 'stop: plan regenerated mid-task'
return None
def run_task(task):
for attempt in range(1, MAX_ATTEMPTS + 1):
budget = Budget()
reason = None
for step in agent.run(task, attempt=attempt):
log_step(task.id, attempt, step)
reason = budget.check(step)
if reason:
break
else:
if task.tests_pass():
return finish(task, 'pending_review')
reason = 'run ended with failing tests'
task = task.with_feedback(reason) # feed the reason into the next attempt
return escalate_to_human(task, summary=agent.summarize()) # stop looping, hand it offEscalating doesn't mean the agent failed. Five minutes of a human pointing it at the right file costs less than a sixth loop, and much less than reviewing the diff that loop would have produced. Tune the thresholds from your audit data. Don't guess them.
8. When cheaper really is cheaper
Moving to a cheaper model isn't wrong in itself. The mistake is usually doing it on the wrong steps. Cheap models do well on work that's deterministic, low-stakes and high-volume, where you can check acceptance automatically.
- ▹Good candidates: formatting, structured extraction into a schema, classification and routing, log summaries, commit messages, ticket tagging.
- ▹Poor candidates: planning, debugging that spans several files, and any step whose output a human has to review closely.
- ▹The rule: downgrade a step only when your eval set from earlier in the course shows its acceptance rate doesn't drop. Run both models on the same set and compare acceptance, not cost.
- ▹Re-run that eval after every model or prompt change. A cheap model that passed last month can regress without anyone noticing.
9. EM corner: rebuild the dashboard
Take 'spend this month' off the headline. As the main number, it rewards cutting the invoice even when that pushes cost onto people. Use these three numbers instead:
- ▹Cost per accepted change, using the full formula with the loaded rate. This is the headline.
- ▹Retry rate: attempts per accepted change, plus the loop share of tokens. This is your leading indicator. It moves before the headline does.
- ▹Reviewer minutes per change. You're already paying this human cost, you just can't see it.
- ▹Raw API spend stays on the dashboard, below the others, as context. It isn't the target.
Finance usually likes these numbers better. They're unit economics: what one unit of output costs, which is how finance already thinks about cost of goods. They grow with output, so spending more because the team ships more doesn't look like waste. They also include engineer time, which in a software team is usually one of finance's biggest line items. 'We want to spend $120 more on API to save 24 reviewer hours a month' is an easy approval. A line that only says 'API spend went up' is not.
10. Bridge to Day 24
You now know which runs are worth paying for, and you've cut the money lost to loops. The next lever is making each token you still pay for cheaper. Day 24 covers context hygiene and prompt-cache-friendly architecture: stable content at the front of the prompt, volatile content at the end, and none of the long re-sent context that made the step-10 re-read so expensive.
Takeaway card
FORMULA
(API + review + rework + cleanup, in $) / changes merged and not reverted within N days
THREE TRACE FIELDS
task_id attempt final_outcome (accepted | rejected | reverted)
MAX-ATTEMPT RULE
Cap attempts (e.g. 3). Stop on: same file read > 2x, same test failing 2x, re-plan.
Escalate to a human with a summary. Never loop silently.
ONE SENTENCE
Optimize the denominator, not the invoice.Extend your knowledge
- ▹Run the Day 23 audit on one week of your own traces and record three baselines: retries per accepted change, loop share of tokens, and reviewer minutes per change. You'll compare against them after Day 24.
- ▹Read Anthropic's engineering post 'Building effective agents', especially the parts on keeping agents simple and on adding stopping conditions and human checkpoints.
- ▹Look up the DORA metrics, change failure rate in particular. 'Not reverted within N days' applies a similar idea to agent output.
- ▹Check the OpenTelemetry GenAI semantic conventions so your task_id, attempt and outcome fields sit alongside the standard token and model attributes rather than in a custom format.
Discussion
Chat with Chi Cong (AI) about this article. Your conversation is private to you — you can publish a summary for others when you're done.