Calibrated doubt in agent PRs

Day 28: Trust is a feature you design for the reviewer

A better model can make your team ship slower. Once coding agents write most of the code, writing it stops being the slow part. The slow part is a tired human deciding whether to press merge. If your agent can't tell that person where to look, upgrading the model won't help, and it can make things worse.

The two-agent week

For one sprint we had two agent setups opening PRs against the same service. Agent A ran on the stronger configuration. When we spot-checked its output, it was right more often, and every PR description sounded equally sure of itself. Agent B looked a bit weaker on paper. But on roughly one PR in five, it added a short section saying which parts it wasn't sure about.

At the sprint review someone put up the merge-time chart, split by agent. B's PRs merged faster. A's sat in review longer. When they did get through, it was usually after a long comment thread, or as a quiet approval nobody wanted to talk about. So the more accurate agent was shipping less. It took us a while to work out why, and the code had nothing to do with it.

Who is the user here?

This series keeps coming back to one move: name the real user, then name what they're shortest of. For agent output, the user isn't the person who wrote the prompt. The prompter just gets a draft. The risk sits with whoever puts their name on the merge. That's the reviewer, and what they lack is attention, not model IQ. A reviewer gets a few hours of real focus a day, and every PR uses some of it. An agent that writes slightly better code but wastes that focus is a worse product for the one person who decides whether the work ships.

Why confidence everywhere costs you

When every part of a PR sounds equally sure, the reviewer can't tell the risky parts from the boring ones. That leaves two options, and both are bad:

• Read everything deeply. It's safe but slow. The PR sits in the queue, the reviewer burns out, and the agent's speed advantage disappears.
• Rubber-stamp everything. That's fast until a bug gets through. After that the team stops trusting every agent PR, good ones included, and goes back to reading everything.

Trust drops either way. Uniform confidence carries no information. To a reviewer it reads as 'I didn't think about where I might be wrong'.

Here's one we lived through. A confident agent PR touched about a dozen files: some renames, a config change, and a small change to how a background job acknowledged messages. The description was neat and sure of itself. The reviewer was at the end of a long day, saw mostly mechanical changes, and skimmed it. Because of the acknowledgement change, some messages could be processed twice under load. It went out and we reverted it. Nobody blamed the reviewer afterwards, because nothing in the PR set that one hunk apart from the renames. Every part of the PR looked equally safe, and one part wasn't.

What calibrated doubt looks like in practice

Here's the shape of the PR descriptions that worked, with details removed. Every claim is specific, and every doubt comes with a reason the reviewer can do something about.

## Summary
Move invoice export from the nightly cron to the job queue.

## Confident (skim is fine)
- export/handler.ts, export/schema.ts, export/mapper.ts
- queue/register.ts, config/jobs.yaml, tests/export.spec.ts
Reason: mechanical move, existing tests cover mapping, new test covers enqueue.

## Unsure (please read closely)
- export/retry.ts, lines 40-78: retry with backoff, max 5 attempts.
  Why: I couldn't find the upstream billing API's timeout contract.
  If their timeout is longer than our 30s backoff ceiling, we may
  double-submit. Who owns billing-api docs?

## Did not read / out of scope
- migrations/2019_legacy_invoice_flags.sql: referenced by mapper,
  not opened. If those flags still matter, mapper.ts line 22 may be wrong.

## How I verified
- Ran export.spec.ts locally (pass). Did not run against staging billing.

Reviewers changed how they worked straight away. They skimmed the confident list, opened the retry file first, and asked the billing team about the timeout. That question is where the real bug turned up. The 'did not read' line became a five-minute check instead of a hidden risk. Reviewers said it felt like working with a colleague who tells you 'I'm not sure about this bit', not one who acts like every part is fine.

The throughput maths, kept simple

Three numbers matter: review time per PR, how many lines get a deep read versus a skim, and merge latency (the time from opened to merged). The numbers below are made up to show how it works. They aren't our data.

• Take a PR with 400 changed lines. A deep read costs about 1 minute per 10 lines. A skim costs about 1 minute per 50.
• Uniform confidence with a careful reviewer: all 400 lines deep-read, about 40 minutes. With a tired reviewer: all 400 skimmed in about 8 minutes, and the risky lines get no more attention than anything else.
• A calibrated PR: 60 flagged lines deep-read (about 6 minutes) plus 340 skimmed (about 7 minutes), roughly 13 minutes in total. The risky lines still get a full deep read.
• Across a whole queue, this is what brings merge latency down. Each PR costs less reviewer time, so the queue clears faster, even though some PRs now arrive with warnings.

So doubt doesn't add review work. It moves it. Attention goes where it's needed, and the reviewer is explicitly allowed to save time everywhere else.

The trap: performative doubt

Once a team sees that doubt helps, the obvious mistake is to prompt the agent to hedge more. An agent that writes 'I'm not fully sure about any of this' is as useless as one that's sure about everything. Neither tells the reviewer anything. Doubt only builds trust when it's calibrated: the flagged parts really are where problems show up, and the unflagged parts really are safe to skim.

We checked this the boring way. For each merged agent PR, we labelled every hunk as confident, unsure or not-read, going by the PR description. A few weeks later we traced reverts, follow-up bug-fix commits and substantive review comments (not style nits) back to those hunks. Then we asked two questions. Where did the problems land? How much of each PR was flagged in the first place?

# per-hunk records: label in {confident, unsure, not_read}, had_problem: bool
from collections import Counter

def calibration(hunks):
    total = Counter(h.label for h in hunks)
    bad = Counter(h.label for h in hunks if h.had_problem)
    rate = {k: bad[k] / total[k] for k in total if total[k]}
    flagged_share = (total['unsure'] + total['not_read']) / len(hunks)
    return rate, flagged_share

# Healthy: problem rate in 'unsure' clearly above 'confident',
# and flagged_share small enough that skimming still saves time.
# Unhealthy: similar rates everywhere (flags are noise), or
# flagged_share creeping up (agent is hedging to cover itself).

Look out for two failure modes. If problems turn up in confident hunks about as often as in unsure ones, the flags are just decoration. If the flagged share keeps growing, the agent has learned that hedging costs it nothing. You fix both in the prompt and the template. Every 'unsure' needs a specific reason, and the agent should be told plainly that flagging everything counts as failing.

Product framing: 'tells me where to look' is a feature

Treat this like any other product decision in the series:

• User: the reviewer who has to approve the merge, not the prompter.
• Job to be done: 'With the little attention I have, help me decide whether this is safe to merge, and show me where my attention matters.'
• Feature: structured confidence, uncertainty that comes with reasons, and an honest list of what the agent didn't read.
• Success metrics: merge latency for agent PRs, plus the revert or bug-fix rate in unflagged code. The second one tells you whether the feature actually works.
• Guardrail metric: the share of each PR that gets flagged. If that creeps up, you're paying for hedging and getting no information back.

Engineers who think in product terms design trust on purpose. They don't wait for a model upgrade to make reviewers comfortable. Reviewers get comfortable when they have information they can act on, not because the code got a bit better.

Monday checklist

• Add three required sections to your agent's PR template: Confident (skim is fine), Unsure (read closely), and Did not read.
• Every 'unsure' needs a concrete reason, such as a missing contract, an untested path or an ambiguous spec. Send back anything that just says 'might have issues'.
• Use reverts and bug-fix commits in unflagged code as your calibration metric. A spreadsheet is fine to start with.
• Tell reviewers in writing that they're allowed to skim the confident sections. Without that permission they'll keep reading everything.
• Once a month, review calibration: problem rate by label, flagged share, and one example of each kind of miss. Change the prompt and template based on what you find.

Bridge to Day 29

Once you're designing for the reviewer, the next question is how you'd know it's working. 'The agent writes more PRs' measures output. 'Reviewed changes ship safely and quickly, and trust still holds months later' measures outcomes. Day 29 is about picking those measures, so the series ends on outcomes rather than output.

Extend your knowledge

• 'Modern Code Review: A Case Study at Google' (Sadowski et al., ICSE SEIP 2018) covers how reviewers actually spend their time and what they look for.
• 'Expectations, Outcomes, and Challenges of Modern Code Review' (Bacchelli and Bird, ICSE 2013) shows that understanding the change is the reviewer's main challenge. Calibrated flags help with exactly that.
• For calibration in forecasting, look up the Brier score and read 'Superforecasting' by Philip Tetlock and Dan Gardner. Both give you words for checking whether stated confidence matches what actually happens.
• Pick ten recent agent PRs from your own repo and label the hunks by hand. Then check where the review comments and fixes really landed. That's your baseline before you change the template.