Evals from real failures

Day 26: Evals come from scar tissue, not spreadsheets

We had 4,000 passing evals the week a customer showed us, in two sentences, that our assistant was making things up. Once an LLM or agent is in front of real users, any change to a prompt, a model version or a tool can quietly break something that used to work. An eval is how you catch that before your users do. It's also far smaller and plainer than the benchmark-and-big-dataset picture most people carry around.

The support ticket

The assistant drafted replies to customer support tickets. It called an order-lookup tool, wrote a response, and a human agent approved it before it went out. Then one week this landed:

> "I asked where my order was and your assistant said it was delivered. It hasn't even shipped."

Our eval dashboard was green that same week. Thousands of generated cases covering tone, format, order lookups, refunds and escalations, all passing. 4,000 tests on our side, two sentences on theirs. And the user was right.

Nothing in the suite was broken. It just tested the wrong things. The rest of this lesson covers why that happens, and the simple habit we adopted to fix it.

What an eval actually is

An eval has three parts: an input, a description of what good looks like, and an automatic check that compares the two. Nothing else. Think of it as a unit test for behaviour that isn't deterministic. A unit test asserts that add(2, 2) == 4. An eval asserts that when the order tool returns no shipment, the reply doesn't claim the order was delivered. The wording can change on every run. The behaviour you care about shouldn't. Everything from earlier in the series feeds in here. The input is your prompt and context (Days on prompting and context), and for agents it also includes whatever tool results the model saw (Days on agents and tool use). An eval freezes one of those situations so you can replay it after every change.

• ▹Input: the exact messages, system prompt version and tool outputs the model received.
• ▹What good looks like: a behaviour, not a sentence. "Does not invent a delivery status" beats "Says: Your order has not shipped yet."
• ▹Check: code that returns pass or fail. No human in the loop.

Why 4,000 generated cases missed it

Most of our suite came from asking an LLM to "generate realistic customer questions about orders" and pairing each question with a mocked tool response. You get a lot of cases fast that way. You also get two biases baked in. Generated cases test what's easy to imagine: clean questions, well-formed data, the happy path with a bit of variation. And they test what's easy to score: fields you can string-match, formats you can validate. Real users and real systems break things in ways nobody thought to imagine.

This side-by-side is what changed my thinking:

# Synthetic case (one of ~4,000). Passed.
input:
  user: "Hi, where is my order #48213?"
  tool_result:
    order_id: "48213"
    status: "shipped"
    carrier: "DHL"
    eta: "2026-04-14"
expect: reply mentions shipped and the ETA

# The real failing case, rebuilt from the ticket. Never in the suite.
input:
  user: "ordered last tues still nothing?? is it coming or not"
  tool_result:
    order_id: "51907"
    status: null          # order created, warehouse hadn't picked it yet
    shipments: []
expect: ???   # nobody had ever written this down

Every synthetic case had a status. The real one had a null, an empty list and a vague, impatient question. Given missing data, the model filled the gap with the most likely ending to an order story: "delivered". No generator was ever going to produce that case. Nobody had asked it to picture our warehouse's data model halfway through an update.

The fix: turn the ticket into a regression case

We didn't respond with another 1,000 generated cases. We turned that one ticket into one eval, following four steps we now use for every incident.

• ▹1. Capture the complaint verbatim. Pull the real trace: system prompt version, user message, exact tool output. Strip personal data (names, addresses, emails, order IDs that map to real people) before anything touches the repo.
• ▹2. Reduce to a minimal input. Cut everything you don't need to reproduce the failure. For us, the null status alone was enough. The vague phrasing only made it worse.
• ▹3. Write the expected behaviour as a rule, not a script. "When there is no shipment record, the reply must not state or imply shipped, delivered or in transit. It must say the status can't be confirmed yet and offer a follow-up."
• ▹4. Pick the cheapest check that actually catches the failure. Then confirm it fails on the current build, before you fix anything.

How to choose the check:

• ▹Exact match: for structured output with one right answer. A JSON field, a classification label, the tool name the agent should call.
• ▹Rule (regex, schema or code assertion): for any failure you can describe as something that must or must not appear. Cheap and deterministic, so make it your default.
• ▹LLM judge: only when the quality you need won't fit in a rule, like "acknowledges uncertainty politely". Keep the judge's question narrow and yes/no.

# evals/cases/order_status_null.py
import re

CASE = {
    "id": "incident-order-status-null",
    "source": "support ticket, postmortem PM-xx",
    "messages": [{"role": "user", "content": "ordered last tues still nothing?? is it coming or not"}],
    "tool_result": {"order_id": "TEST-0001", "status": None, "shipments": []},
}

FORBIDDEN = re.compile(r"\b(delivered|shipped|in transit|on (its|the) way|out for delivery)\b", re.I)

def check(reply: str, judge) -> bool:
    # Rule: never invent a status the tool didn't return
    if FORBIDDEN.search(reply):
        return False
    # Narrow LLM judge for the part a regex can't see
    return judge(
        question="Does the reply clearly say the order status cannot be confirmed yet "
                 "and offer a next step? Answer yes or no.",
        reply=reply,
    )

The rule carries most of the weight. The judge handles the one thing a regex can't see, and since its question is narrow, it's easy to trust. We ran each case several times per change, because a non-deterministic model can pass once by luck. A case only counted as passing if it passed every run.

Six months later: 40 vs 4,000

We kept feeding the scar-tissue suite one case per incident, near miss or reviewer catch. Six months in, it held roughly 40 cases. The generated suite was still sitting around 4,000. Over that period we had prompt edits, two model upgrades and a tool schema change. Here's the honest tally of which suite flagged the resulting regressions:

• ▹The small suite flagged most of the regressions that would have reached users, and usually got there first. No surprise: they tended to be old failures coming back in a slightly different shape, which is exactly what the small suite holds.
• ▹The big suite earned its keep on broad, shallow breakage. After one model upgrade, output formatting drifted and a JSON field got renamed. It caught both straight away. The small suite covered neither.
• ▹A few regressions slipped past both and turned up as new tickets. Each one became case 41, 42 and so on. That's the system working as intended.
• ▹The big suite was also noisy. Some failures came from cases whose "expected" answer was itself wrong. Nobody owned those cases, so nobody fixed them, and people learned to ignore red.

That last one is the real lesson. A suite nobody owns turns into an abandoned notebook. Each of the 40 cases had a name and a story behind it, so when one went red, somebody cared.

The routine, not the tool

There's no shortage of eval tooling: promptfoo, OpenAI Evals, Inspect, LangSmith, Braintrust, or a plain pytest file. The tool isn't what makes this work, though. The routine is. Treat incident → eval the way mature teams already treat incident → postmortem.

• ▹Who owns it: whoever owns the incident. Writing the postmortem means writing the eval. Add an "eval case" field to your postmortem template so nobody can quietly skip it.
• ▹When it gets written: before the fix ships. Watch it fail, apply the fix, watch it pass. Write it afterwards and you've never proved the check would have caught anything.
• ▹Where it lives: in the repo, beside the prompt or agent code it protects (evals/cases/, say), reviewed in the same PR.
• ▹When it runs: on every change to a prompt, model version, tool schema, retrieval config or agent instructions. If coding agents are editing your prompts for you, this is the only thing stopping them from undoing a lesson you've already paid for.

What this does NOT mean

Generated and benchmark evals still have value. They're broad smoke tests: cheap to produce, good at catching format drift, schema breaks and big capability drops after a model swap, and handy for comparing models before you have any real traffic. Where they fall short is depth. They rarely contain the specific, odd failures your product actually has. Scar-tissue evals are the opposite. Narrow, expensive to collect (each one cost you an incident), and very sharp. Run both. A wide, shallow generated layer catches obvious breakage. A small, deep layer of real failures stops old bugs coming back. If you can only maintain one, keep the 40 real ones.

Your Day 26 exercise

Pick one bad output you've seen this week from any AI tool. A coding agent that deleted a test, a chatbot that made up a policy, a summary that dropped the one number that mattered. Write it down in three lines:

input:    <the exact prompt/context that produced it>
expected: <the behaviour you wanted, written as a rule>
check:    <exact match | rule | LLM judge, and what it looks for>

Congratulations, that's your first eval. If you can, re-run the input a few times and see whether your check would have caught the failure. Can't run it automatically yet? Fine. Writing it down is the step most teams skip.

Next: Day 27

Today we used an LLM judge for exactly one narrow question. As your suite grows you'll lean on judges more, and that raises an uncomfortable question: what happens when the thing grading your model is itself a model? Tomorrow covers when you can trust an automated judge, when you can't, and how to check the checker.

Extend your knowledge

• ▹Read Hamel Husain's writing on LLM evals, especially his push to look at your data and build evals from error analysis rather than generic metrics.
• ▹Port your Day 26 exercise into a runnable case with promptfoo or Inspect (from the UK AI Security Institute). One rule check, and at most one judge question.
• ▹Read the postmortem culture chapter in Google's SRE book, then add an "eval case" field to your team's incident template.
• ▹Go through your last 10 support tickets or agent PR review comments and count how many could become a regression case. That number is your backlog.