Cost per accepted change

Day 30: Cost per accepted change, the number that actually matters

Most teams pick a model the way they'd pick a cloud instance: open the pricing table and sort by cheapest. A quarter later the AI budget is still climbing, delivery is flat, and nobody can say why. The reason is simple. You don't ship tokens. You ship changes that get merged and stay merged. Price per token is a real number, but it's the wrong one to manage by. This last lesson of the series gives you the right one.

The hook: two invoices for one fix

Here's the task: fix a validation rule. An email field accepts addresses with a trailing dot, and it shouldn't. You give the same task to two models. This is everything the pricing table tells you:

                 Input (per 1M tokens)   Output (per 1M tokens)
Model A          $1                      $5
Model B          $3                      $15

Model A is 3x cheaper per token.

Model B gets the fix right on the first try. Model A takes three attempts, and an engineer has to finish the job. So which one was cheaper overall? Make a guess and hold on to it. We'll work out the real answer further down. (These prices are illustrative. Check your provider's current pricing page for the real numbers.)

What you're actually billed for

An LLM invoice has three main lines, each with its own price:

• Input tokens: everything you send the model on a call. That means the system prompt, tool definitions, conversation history, file contents and tool results. (Day 2: tokens are the unit the model reads and writes in.)
• Output tokens: what the model writes back, whether that's reasoning, code or tool calls. Per token, output costs several times more than input. The multiple depends on provider and model; somewhere around 4–8x is common.
• Cached input tokens: input the provider has already seen as an identical prefix on a recent call. On Anthropic's API, a cache read costs about a tenth of the normal input price. Writing to the cache costs more than normal input: 1.25x for the default 5-minute cache, 2x for the 1-hour option. (Day 4: the context window is everything the model sees on a single call.)

Here's what matters most: in an agent loop, most of your bill is input. Very little of it pays for new writing. You're mostly paying for the agent to re-read the same context, once per step. Output costs more per token, but there's far less of it.

Why the agent loop multiplies the bill

Back in the agents and tool-use lessons, you saw that the model has no memory between calls. So every step of the loop re-sends the whole conversation so far: system prompt, tools, task, every earlier tool call, every tool result. The next step then adds more on top. Context grows each step, so each extra step costs more than the one before.

Take a 6-step loop. It starts at 10k tokens (system prompt + tools + task), and each step adds roughly 5k (a file read, a test run, the model's own output):

Step   What happens              Input tokens sent   Output tokens
1      read task, plan           10,000              1,000
2      read validator file       15,000              1,000
3      read tests                20,000              1,000
4      edit code                 25,000              1,000
5      run tests                 30,000              1,000
6      summarise, open PR        35,000              1,000
-----------------------------------------------------------------
Total                            135,000             6,000

At Model B prices ($3 in / $15 out):
  input  = 135,000 x $3/1M  = $0.405
  output =   6,000 x $15/1M = $0.090
  input is ~82% of the bill

Now the tests fail, and the agent retries in the same session. Steps 7–12 don't start fresh. They start from a context that's already 35k+ tokens and keep growing: 40k, 45k, 50k, 55k, 60k, 65k. That's another 315k input tokens, and total input goes from 135k to 450k, about 3.3x. So a retry doesn't just double the cost. It's stacked on top of everything that came before, and it can more than triple it.

def loop_input_tokens(start=10_000, growth=5_000, steps=6):
    return sum(start + growth * i for i in range(steps))

first_try = loop_input_tokens(steps=6)    # 135,000
with_retry = loop_input_tokens(steps=12)  # 450,000
print(with_retry / first_try)              # ~3.3

The full ledger (illustrative worked example)

Now back to the two invoices. Every number below is invented and rounded so you can check the arithmetic by hand. Two assumptions: one CI run costs $0.50, and an engineer's loaded cost (salary, overhead, benefits) is $120/hour, which works out to $2/minute.

MODEL A  ($1 in / $5 out)
  Attempt 1:  1.0M input, 50k output   -> CI fails
  Attempt 2:  1.5M input, 50k output   -> CI fails (context keeps growing)
  Attempt 3:  2.0M input, 50k output   -> CI passes, misses an edge case
  Tokens:     4.5M x $1/1M + 150k x $5/1M   = $4.50 + $0.75 = $5.25
  CI:         3 runs x $0.50                = $1.50
  Engineer:   write task          5 min
              re-prompt twice    10 min
              review rounds    2 x 20 min = 40 min
              rescue edge case   30 min
              total              85 min x $2 = $170.00
  TOTAL                                       $176.75

MODEL B  ($3 in / $15 out)
  Attempt 1:  1.0M input, 50k output   -> CI passes, review approves
  Tokens:     1.0M x $3/1M + 50k x $15/1M   = $3.00 + $0.75 = $3.75
  CI:         1 run x $0.50                 = $0.50
  Engineer:   write task 5 min + review 15 min = 20 min x $2 = $40.00
  TOTAL                                       $44.25

Model A was 3x cheaper per token, and its change still cost about 4x more. Look at two things. Even on tokens alone, A lost ($5.25 vs $3.75), because every retry re-read a bigger context. And next to engineer time, the token bill barely registers. For both models in this example, people are over 90% of the cost.

Where the hidden costs come from

• Retries. Each one re-sends a bigger context than the last, so cost climbs faster than the attempt count.
• Self-review loops. A "now check your work" step does help quality, but it's another full re-read of the context. A weaker model may go round several times before it's confident.
• Vague task specs. Say the task is just "fix email validation". The agent burns tokens exploring the codebase to work out what you meant: which validator, which rule, which tests. All that exploration lands in the context and gets re-read on every later step.
• Reverted commits. A change that merges and gets reverted next week costs everything it took to make, plus the revert, plus the redo, and it delivered nothing.
• Human rescue time. These are the minutes an engineer spends re-prompting, reviewing a second time, or giving up and finishing by hand. You won't find them on the token invoice, and they're usually the biggest line.

The better unit: cost per accepted change

cost per accepted change =

   token spend + CI spend + (engineer minutes x rate)
   ---------------------------------------------------
   merged PRs not reverted within N days

• Token spend: the dollars your provider charged for all agent work in the period, failed attempts and abandoned sessions included. Count dollars, not tokens, so cached and uncached input get weighted correctly.
• CI spend: compute for every pipeline run the agent triggered, failed runs too. If you can't price it exactly, a flat per-run estimate is fine.
• Engineer minutes × rate: every bit of human time on the change, whether writing the task, re-prompting, reviewing, fixing or reverting. Use the loaded hourly rate (what the person actually costs the company), not take-home pay.
• Merged PRs not reverted within N days: the denominator only counts work that shipped and stayed shipped. Choose N to fit your release rhythm; 14 days is a sensible starting point. Abandoned and reverted PRs still add to the cost but never count as output, and that's deliberate. It's the whole point of the metric.

Use it to compare models, to compare agent setups, to decide whether a type of task is worth handing to an agent at all, and in budget conversations with finance. Don't use it to judge an individual engineer, or to compare very different kinds of work. A typo fix and a migration are not the same "change". Keep comparisons within similar task types.

What we saw at PhoenixDX

We'd been watching the token bill, and it was going down. On the dashboard that looked like a win. Then we noticed shipping had slowed down over the same period. PRs were sitting in review longer, and engineers were spending their days steering and patching agent output instead of doing their own work. We'd been optimising the number on the invoice, and the cost hadn't gone away. It had moved somewhere the invoice can't see, into people's time. That's when we stopped asking "what does this model cost per token?" and started asking "what does a finished change cost?"

Two cheap levers that don't require switching models

Lever 1: a stable, cache-friendly prompt prefix. Prompt caching matches on an identical prefix. Change a single byte near the top and everything after it misses the cache, so it's billed at full input price (or the higher cache-write price) on every step of every loop. Teams break their own cache in a few familiar ways:

• A timestamp, request ID or "current time" at the top of the system prompt.
• A tools list built from an unordered dict or set, so the order shifts between calls.
• Per-user or per-task details inserted into the system prompt instead of appended later.
• Earlier messages in the history edited instead of new ones only appended.

BAD (cache breaks every call)        GOOD (stable prefix, varying tail)
---------------------------------    ---------------------------------
Now: 2026-10-27T09:14:03Z            [system prompt - never changes]
[system prompt]                      [tools - fixed order]
[tools - random order]               [repo conventions - stable]
[task]                               [task + timestamp, if needed]

Lever 2: a tighter task spec before the agent starts. Spend five minutes writing down what "done" means, and the agent doesn't have to explore its way there, then re-read that exploration on every later step. A minimal template:

## Task
Reject email addresses with a trailing dot in the domain.

## Where
src/validation/email.ts, function validateEmail

## Done means
- "a@b.com." is rejected with error code EMAIL_INVALID
- existing tests in email.test.ts still pass
- new test cases added for trailing dot

## Out of scope
Do not change the error message copy or the form UI.

Your exercise: price your last 10 agent-assisted PRs

Pick 10 recent PRs where an agent did a meaningful share of the work, and estimate the four numbers for each. Rough is fine. You're after the order of magnitude, not the cents.

| PR  | Token $ | CI runs x $ | Eng min x rate | Merged? | Reverted <14d? | Counts? |
|-----|---------|-------------|----------------|---------|----------------|---------|
| #   |         |             |                |  Y / N  |     Y / N      |  1 / 0  |
| #   |         |             |                |         |                |         |
| ... |         |             |                |         |                |         |
|-----|---------|-------------|----------------|---------|----------------|---------|
| SUM |         |             |                |         |                |  = D    |

Cost per accepted change = (sum of token $ + CI $ + eng $) / D

If you can't get the token spend for a single PR, write that down, because it's a finding in itself. It means your agent runs aren't tagged by task, and you're managing a budget you can't see. Fix that first (tag each session with a ticket or branch ID) before you compare any models.

Course capstone: how 30 days add up to one number

• Tokens are the billing unit, and input, output and cached input are each priced differently. That's where the numerator starts.
• Context windows: the agent re-reads its whole context every step, so context size drives input cost. A bloated context makes every step more expensive.
• Prompting: a clear spec and a stable prefix cut exploration tokens and turn re-reads into cache hits.
• Tools: every tool result goes into the context. If a tool returns 5k lines when 50 would do, you pay for that on every later step.
• Agents: the loop multiplies all of the above, which is how one retry can more than triple a task's cost instead of doubling it.
• Evaluation: "merged and not reverted" is your eval. Without it you can measure what you spent, but not what you got for it.

So after this course, measure what a finished piece of work costs, not what a token costs. Per-token prices have kept falling, and I expect they'll keep falling. Cheaper tokens won't make your shipped changes cheaper on their own, though. That part depends on how you use everything from the last 30 days.

Extend your knowledge

• Read your provider's prompt caching docs, for example the "Prompt caching" page in Anthropic's API documentation. Write down the minimum cacheable length, how long the cache lives, and the read and write price multipliers.
• Pull usage data with cache-read and uncached input split out (the Anthropic Console usage view, or the usage fields on each API response), then work out your cache hit rate for one agent workflow.
• Look into the DORA metrics from Accelerate by Forsgren, Humble and Kim, especially change failure rate. "Not reverted within N days" is a similar idea applied to agent output.
• Once the two levers (stable prefix and tighter specs) are standard practice, run the 10-PR exercise again a month later and compare the two numbers.