AI coding benchmarks

Day 27: Benchmarks — what a leaderboard score really tells you, and how to build one that fits your team

"Model X scores 72% on SWE-bench." You've seen that headline, or one like it, and sooner or later someone will ask you to choose a model because of it. The number is probably accurate. It just isn't about your codebase. Today we'll look at what these scores really measure and how to get one that tells you how a model will do on your own work.

The hook: an accurate score that was useless to us

We once switched our coding agent to whatever model was top of a public leaderboard. In its first week it opened a PR that brought back a bug we'd reverted a few months earlier. Same mistake, same file, and a commit message just as confident as the human one. The score wasn't a lie. The model really did solve that share of the benchmark's tasks. But those were someone else's tasks. They couldn't tell us anything about our codebase, our weird edge cases, or the traps our team had already fallen into. That's today's lesson: a score can be correct and still be the wrong number for you.

What a benchmark actually is

A benchmark is a fixed exam with an answer key. You get a set of questions or tasks, each with a known correct outcome, and a rule for marking answers right or wrong. A model's score is its result on that exam. That's all it is. Compare it to a driving test. Passing proves you can handle that route with that examiner. It doesn't prove you can drive through your city at rush hour, in the rain, on roads you've never been on. Earlier in the course we covered evaluation in general: define the task, define what "good" means, measure it the same way every time. A benchmark is one kind of evaluation. It's frozen and shared, and lots of people run it so their results can be compared. The frozen, shared part is what makes it useful. It's also what makes it fragile.

How to read a headline number

Start with the most common headline in AI coding: "Model X scores N% on SWE-bench." SWE-bench is built from real GitHub issues in open-source Python projects. Each task gives the model the issue text and the repository as it looked before the fix, and the model has to write a patch. The patch is then checked against tests taken from the real fix. Tests that failed before the fix have to pass, and tests that already passed have to keep passing. SWE-bench Verified is a smaller subset that humans reviewed to remove tasks that were ambiguous or couldn't be solved.

"% resolved" is the share of tasks where the model's patch passed those tests. It says nothing about whether the code is readable, whether a reviewer would approve it, or whether a senior engineer would have fixed it that way. It only tells you whether the hidden tests went green.

Before you trust any score, ask three questions:

• ▹On what tasks? Which benchmark, which subset, which languages, what kind of work? A Python bug-fix benchmark tells you very little about your TypeScript monorepo, and even less about your Terraform.
• ▹Compared against what? Which other models, and run with which agent scaffold (the tools, prompts, and loop wrapped around the model)? The same model can score very differently in different harnesses, so a headline number often measures the model and the harness together.
• ▹At what cost, and how many attempts? Was it one try (pass@1) or the best of several? How many tokens, minutes, and dollars per task? If a score needs 5 attempts and a large token budget per task, it's describing a different product from a one-shot score.

Why scores drift away from reality

Reason 1: contamination. Public benchmarks are built from public data like GitHub issues, merged PRs, and Stack Overflow threads. Models are trained on huge scrapes of that same public data, so the exam questions can end up in what the model studied, and sometimes the answers do too. A high score might mean the model remembered the answer rather than working it out. Agents add a newer version of the problem. An agent with web search or full repository access can sometimes find the real fix during the test, for example by reading the upstream commit or a discussion thread. At that point you're testing its searching, not its reasoning. Benchmark maintainers try to prevent both, but you're trusting that they did it well.

Reason 2: mismatch. Benchmark tasks are chosen to be fair, which usually means a clear issue description, a bug you can reproduce, and tests that settle pass or fail. Your backlog doesn't look like that. Tickets say "export is broken for some customers, see Slack." The real constraint is in someone's head, and the tests that matter haven't been written yet. Your codebase has conventions, internal libraries, and history the model has never seen. A model can handle tidy tickets well and still struggle with messy ones, and a benchmark has no way to show you that.

The fix: your reverted PRs are a private exam

Every team already has a set of good test cases, usually without realising it: its reverted PRs. A revert means a real engineer took a real ticket and wrote a change that looked fine. It got through review and CI, shipped, and then had to be undone. That's exactly the kind of task you want an AI agent to handle better than a human on a bad day.

• ▹They're real. Your tickets, your code, your conventions, not polished open-source issues.
• ▹You've already paid for them. Every revert cost an incident, a rollback, or a bad day. "Would this model make the same mistake?" is the most direct question you can ask.
• ▹The answer key comes with them. You have the bad change, the revert, and usually a later fix that got it right. That's enough to write a check.
• ▹They almost certainly aren't in any training set. Private repo history isn't in public scrapes, and an agent can't look the answer up online. The contamination risk stays low as long as the repo stays private and none of your tool vendors are allowed to train on your code. Go and check those settings.

Step by step: build it in one afternoon

Step 1: find about 30 reverts from last quarter. For most teams, git is all you need. Revert commits usually start with "Revert" and reference the original commit.

# List revert commits from the last 90 days
git log --since="90 days ago" --grep='^Revert' --format='%h %ad %s' --date=short

# For each revert, find the original commit it undid
git show <revert_sha> | grep -i 'This reverts commit'

# The starting point for the exam is the parent of the bad commit
git rev-parse <bad_sha>^

Skip reverts that were only about infrastructure or timing, like "reverting to unblock the release train," where the code was fine. You want the ones where the code itself was the problem.

Step 2: write each case down. For every revert, copy the original ticket text exactly as it was written, vagueness included. Record the commit to start from (the one before the bad change). Then add a check that fails on the bad change and passes on the correct fix. Often that check is simply the regression test someone added with the later fix.

# cases/case-017.yaml
id: case-017
ticket: |
  Add pagination to GET /orders. Page size 50. Mobile app needs it this sprint.
base_commit: 3f9c2ab        # parent of the reverted change
bad_commit: 7e1d044         # what a human shipped (kept for reference only)
check:
  command: pytest tests/regression/test_orders_pagination_ties.py
  must_pass: true
notes: |
  Original bug: sorted by created_at only; rows with identical timestamps
  were duplicated or skipped across pages. Revert in prod after 2 days.

Step 3: run every candidate model or agent on every ticket. Each run starts from a fresh checkout of base_commit, gets only the ticket text, and has the same tools as every other run. Don't show it the bad commit, the revert, or the check. Turn off web search, or at least make sure the agent can't reach your Git host's history past base_commit.

Step 4: score whether it avoided the bug, not whether it compiled. Building and passing the existing tests is the minimum. The human's bad change cleared that bar too, and it still got reverted. A case only passes if your regression check passes.

# Minimal scoring loop (sketch)
import subprocess, yaml, pathlib

BUDGET = {"max_tokens": 200_000, "max_attempts": 1, "timeout_min": 20}

for case_file in sorted(pathlib.Path("cases").glob("*.yaml")):
    case = yaml.safe_load(case_file.read_text())
    for agent in ["model_a", "model_b", "model_c"]:
        workdir = fresh_checkout(case["base_commit"])           # your helper
        run = run_agent(agent, case["ticket"], workdir, BUDGET)  # your helper
        ok = subprocess.run(case["check"]["command"], shell=True, cwd=workdir).returncode == 0
        record(case["id"], agent, passed=ok,
               tokens=run.tokens, attempts=run.attempts,
               asked_question=run.asked_clarifying_question,
               diff=run.diff)

What to look for beyond pass or fail

With only 30 cases, the totals tell you less than how each run actually went. For every run, write down:

• ▹Did it ask a clarifying question? A lot of reverts start with a vague ticket. An agent that asks "what should happen to cancelled orders?" is showing the judgment you want, even if your harness counts the question as a non-answer. Give it its own column.
• ▹Did it make the same mistake the human made? Compare its diff to bad_commit. If it made the same wrong assumption, the trap is in the ticket or the codebase, not only in the person who fell into it. That's worth knowing whichever model you pick.
• ▹Tokens, retries, minutes. A model that passes 20 of 30 at a third of the cost might be a better choice than one that passes 22.
• ▹Was it a fair fight? Every model gets the same token cap, number of attempts, tools, and timeout. If one agent gets 5 retries and another gets 1, you're comparing budgets, not models.

Worked mini-example: one revert, end to end

The ticket (anonymised): "Add pagination to GET /orders. Page size 50. Mobile app needs it this sprint." What a human shipped: offset pagination ordered by created_at. It passed review and CI. In production, though, bulk imports create lots of orders with the same created_at timestamp. Rows with equal timestamps came back in no fixed order, so some orders appeared on two pages and others never appeared at all. The change was reverted after two days. The later fix ordered by (created_at, id), so every row has a unique position, and switched to cursor-based pagination.

The check: a regression test that inserts 120 orders with the same timestamp, pages through all of them, and asserts that every ID shows up exactly once.

def test_pagination_with_timestamp_ties(client, make_orders):
    make_orders(count=120, created_at="2026-07-01T00:00:00Z")
    seen, cursor = [], None
    while True:
        page = client.get("/orders", params={"cursor": cursor}).json()
        seen += [o["id"] for o in page["items"]]
        cursor = page.get("next_cursor")
        if not cursor:
            break
    assert len(seen) == len(set(seen)) == 120

A fail: Model A adds LIMIT/OFFSET with ORDER BY created_at and writes a tidy unit test using orders with distinct timestamps. Every existing test passes. It's the human's mistake again, nearly line for line. The regression check finds duplicates on page 2. Fail, tagged "repeated original mistake."

A pass: Model B orders by (created_at, id) and returns a cursor built from both values. Model C stops first and asks: "Can created_at values collide, for example from bulk imports? If so I'll add a tie-breaker." Both pass the check once the question is answered. Model C also gets the "asked clarifying question" flag, which counts in its favour when your tickets are as vague as most tickets are.

No public leaderboard would have shown you that difference. Your own case set showed it after one run.

Limits, stated honestly

• ▹30 cases gives you a signal, not proof. A one- or two-case gap between models is noise. A gap of eight or ten is worth acting on. Use the results to narrow your shortlist, then confirm with a short trial on live work.
• ▹It goes stale. Your codebase moves on, old base commits stop building, and new kinds of failure show up. Every quarter, add that quarter's reverts and retire cases that no longer match how you work.
• ▹Keep it private. Once these cases end up in a public repo, a blog post, or a vendor's eval portal that might train on them, they can get contaminated like any public benchmark. Treat the set like internal source code.
• ▹Reverts only show you the bugs that got caught. Failures that slipped through quietly aren't in there. That's fine for comparing models, but don't mistake the set for a full picture of quality.

Day 27 homework

• ▹Build a 10-case version this week. Pick 10 reverts where the code was the problem, write a YAML file for each (ticket, base commit, check), and run at least two models or agents on the same budget.
• ▹Write down the three questions (on what tasks, compared against what, at what cost or with how many attempts) and use them on one leaderboard claim you've seen this month.
• ▹Bring your results to Day 28: pass rate, tokens per case, how often each model repeated the original mistake, and how often it asked a clarifying question.

Extend your knowledge

• ▹Read the original SWE-bench paper (Jimenez et al., "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?") and OpenAI's write-up on SWE-bench Verified to see how the tasks are built and filtered.
• ▹Look at contamination-aware benchmarks like LiveCodeBench, which keeps collecting new problems and lets you score a model only on problems published after its training cutoff. It's the same idea as your private set, done in public.
• ▹Read Hamel Husain's writing on building evals from your own product's real failures. It's the general version of today's reverted-PR technique.
• ▹Turn your 10-case set into a CI job that runs whenever you're thinking about switching model versions, so no model change goes in without passing your own exam first.