The 94% That Was Lying: How Our Eval Harness Quietly Stopped Testing Reality

Day 14: Your Eval Harness Is a Second Production System

You shipped an eval suite. It's green. You trust it. Stop — that trust is the bug. An eval harness is just code, running against a target that keeps moving, and code nobody owns rots exactly like any other unowned service: quietly, until the day it starts lying to you outright.

The mystery: 94% and flat, while complaints tripled

On one of our agents, the eval dashboard had read 94% for three releases running. Green, stable, boring — exactly what you want from a metric. In that same window, support tickets tagged "agent gave wrong answer" tripled. Two systems, same product, telling opposite stories. Either the eval was wrong or the users were, and honestly, we already knew which one it was going to be. We still didn't want to open the file.

The autopsy: a harness stuck in month one

So we opened it. Every fixture was a recorded prompt/tool-call/response triple, captured from the very first version of the agent — original tool schema, original system-prompt format. Since then the agent had shipped four versions: two tool renames, one schema migration (arguments moved from flat fields into a nested object), and a rewritten system prompt. The harness had registered none of it.

fixture.json (written month 1, never touched again):
{
  "tool_call": "lookup_order(order_id)",
  "expected": "tool_call"
}

agent v4 actually emits:
{
  "tool_call": "orders.lookup({ id: ... })"
}

-> harness marks this a MISS on tool-call matching...
-> ...except the judge prompt was ALSO written month 1,
   and it does fuzzy semantic matching that quietly waves
   this through as "close enough."
-> net effect: false pass, for the wrong reason.

That last line is the part that stings. It's not that the harness failed loudly on the new schema — an unrelated leniency in the judge prompt just happened to paper over the mismatch. The green checkmark wasn't earned. It was an accident.

Name the pattern: eval harnesses are production systems

A production system is anything with dependencies that shift under it, a deploy surface, and a decay curve the moment you walk away. Your eval harness has all three. It depends on the agent's tool schema and prompt format. It "deploys" every time someone edits a fixture or a judge prompt. And it decays the instant the agent moves and the harness doesn't follow. The only thing missing is a line on the org chart. Nobody names an owner for it, because by the time it exists, the team has already filed it under "shipped, done, box checked" — the exact classification that lets a database migration or an auth service rot too, once you stop watching it.

The three decay modes we actually found

• ▹Stale fixtures — recorded inputs and outputs frozen at whatever tool schema and prompt format existed the day someone first wrote the test. Every agent version shipped since is being graded against a contract that no longer exists.
• ▹A judge prompt nobody re-validated — the LLM-as-judge instructions were written once, against one set of failure modes, and never run again against how the agent fails now. It got looser over time by accident, not by design, and started scoring near-misses as passes.
• ▹Fixtures pinned to dead tool calls — several test cases still asserted on tool names and argument shapes renamed two releases ago. Those cases couldn't fail anymore even in principle. They were just collecting a paycheck on the metric.

Why the green checkmark survived so long

Because the eval was measuring agreement with its own past self, not agreement with the user. Every fixture encodes somebody's judgment call, frozen at one point in time, about what a correct response looks like. If the agent, the schema, and the judge prompt all drift together — or the judge prompt drifts just enough to quietly compensate for changes it was never told about — the score holds flat by construction. A steady 94% doesn't mean the agent is fine. It can just as easily mean the ruler stopped moving. The one signal that isn't self-referential is the user — and that's exactly the signal that had been tripling in the ticket queue the whole time.

What we changed

• ▹Version the harness alongside the agent. Every agent release tags a matching harness commit; a fixture written against v2's schema is labeled v2 and either migrates or gets retired when v3 ships — it doesn't just keep running against whatever's current and hoping for the best.
• ▹Someone is on-call for eval drift. Not the team in some vague, implicit sense — one named owner whose job includes noticing when the pass rate and user-reported quality stop agreeing, the same way an SRE watches a dashboard for a service they didn't personally write.
• ▹Every test case gets a deprecation date, like any other code. When you write a fixture, you also write down what would invalidate it — a schema version, a prompt version, a tool name — so it expires loudly instead of quietly turning into a no-op.

The uncomfortable generalization

If you can't answer "who owns the eval harness right now, and when did they last touch the judge prompt" — you already have this bug. You just haven't measured it, because the instrument you'd use to measure it is the thing that's broken. This isn't some quirk of agentic systems specifically: the faster you iterate on prompts, tools, and schemas, the faster the harness falls behind, because nothing forces the two to move in lockstep the way a database migration forces an app to update its queries.

Day 15: now that you know the harness can lie to you, next lesson is how to build one that catches its own drift — instrumenting the eval system itself so staleness shows up as a signal instead of a silent green light.

Extend your knowledge

• ▹Audit your own harness this week: git-blame every fixture file and the judge prompt. If the last touch predates your last two agent releases, you already have this bug — you just haven't measured it.
• ▹Read Hamel Husain's writing on LLM evals and error analysis. He's sharp on why evals need continuous re-validation, not a one-time build you check off and forget.
• ▹Look at how OpenAI's `evals` framework treats eval definitions as versioned code with an explicit registry, not a static fixture folder — worth stealing even if you don't adopt the whole framework.
• ▹Set up a lightweight drift check: sample production traces monthly and re-run them through your judge prompt to see if its verdicts still match human judgment on real inputs, not just fixtures.