Your Eval Score Sat at 94% for Three Months. That Was the Problem.

Why This Matters

An eval score is only as honest as the labels behind it. Stop checking those labels, and your dashboard can sit on green for months while the thing it's supposed to protect rots quietly underneath.

The Dashboard Said 94%. The Tickets Said Otherwise.

Here's a dashboard I'd bet you've seen some version of. An LLM support agent, 94% pass rate, flat and green for three straight months. Every release note reads 'no regression.' Meanwhile the 'wrong answer from bot' ticket tag is climbing, month over month. Nobody connects the two, because the one system built to catch exactly this — the eval suite — keeps insisting everything's fine. That contradiction is the whole lesson: a stable score and stable quality are not the same claim.

What Turned Up When Someone Actually Opened the Dataset

When the team stopped running the eval and started reading it — line by line, example by example — they found a dataset answering questions about a product that no longer existed. Nobody had touched it since the day it was written.

• ▹Self-serve refunds had shipped months earlier. An entire category of 'golden' denial answers was now just wrong.
• ▹One edge case — invoice attached, partial refund requested — used to be rare enough to earn a single example in the set. After a pricing change it became the most common ticket type in the queue. The eval still treated it like a footnote.
• ▹The team had swapped the underlying LLM. The new model was more concise and more willing to ask a clarifying question before answering — genuinely better behavior. The old labels scored it as a failure anyway.

Notice what none of this touched: the scoring pipeline. The harness ran fine. The judge model, the exact-match logic — all working exactly as designed. It was faithfully grading against a target that had quietly moved out from under it.

Name the Failure Mode: A Golden Dataset Is an Opinion With a Timestamp

Here's the mental model to fix in your head: a golden dataset isn't ground truth. It's a snapshot of what your team believed 'correct' meant on the day someone wrote the label. Ground truth keeps moving. The snapshot doesn't — not unless someone makes it.

Make it concrete. Month 1: someone asks 'Can I get a refund for a canceled subscription?' The label says 'No, refunds are not available for canceled subscriptions.' Correct, matches policy — the eval scores every matching answer as right. Month 3: the refund policy changes. The correct answer is now 'Yes, if canceled within 14 days.' Your model, up to date, gives that answer — and gets marked wrong, because nobody touched the label. Flip it around and it's worse: a model still parroting the old policy gets marked right. The eval can't tell 'the model is correct' apart from 'the model agrees with a stale label.' Those two things are only the same claim on day one.

Why This Is Worse Than Having No Eval Set At All

No eval set at all keeps you honest, in a way. You know you're flying without instruments, so you test by hand, you watch the ticket queue, you ship carefully. A stale eval set does the opposite — it manufactures confidence you haven't earned. You tune the prompt, the retrieval, the fine-tune, watch the number climb, and call it progress, even though the target it's climbing toward stopped matching reality months back. In an agent pipeline this compounds fast: an agent tuned to please a dead rubric can be actively getting worse for real users while every dashboard you own says it's getting better.

The Fix: Treat the Dataset as a Versioned Claim, Not a Fixed Artifact

So stop writing the golden dataset once and defending it forever. Treat every example as a claim — tied to a specific version of the product, the policy, the model — and every claim expires unless someone actively renews it.

• ▹Tag every example with the spec/policy/model version it encodes. A label with no version attached is a label you can't trust — you have no idea what it was true against.
• ▹Timestamp when it was last reviewed, not just when it was written. Creation date tells you nothing about whether it still holds.
• ▹Give the dataset its own review cadence, separate from your eval run cadence. Running the eval every night doesn't mean anyone's looked at the labels this month.
• ▹Put a name on it. Someone — or a rotation — owns re-litigating labels, the same way someone owns on-call for a production incident.
• ▹Report 'the score went up' and 'the labels got reviewed' as two separate facts. Never let one stand in for the other.

This Week: Run a Label-Age Audit

You don't need new tooling for this. Add two fields to every example in your golden set, and run one report.

# one entry in golden_set.yaml
- id: refund-canceled-sub-001
  input: "Can I get a refund for a canceled subscription?"
  expected: "Yes, if canceled within 14 days."
  spec_version: policy-v3-2026-08
  last_reviewed: 2026-08-14
  reviewed_by: cong.chi@phoenix-dx.com

# flag anything untouched past N weeks
from datetime import date

STALE_AFTER_WEEKS = 6

def audit(golden_set, today=date.today()):
    stale = []
    for ex in golden_set:
        age_weeks = (today - ex["last_reviewed"]).days / 7
        if age_weeks > STALE_AFTER_WEEKS:
            stale.append((ex["id"], round(age_weeks, 1)))
    return stale

Put that audit's output right next to your eval score, in whatever dashboard or standup you already run. 'Eval: 94%, 0 stale labels' is a number you can trust. 'Eval: 94%, 31 stale labels' is a warning sign wearing a green light.

Where This Goes Next

Notice everything above assumed one model, graded against one dataset. That's the easy version of this problem. Tomorrow we look at what happens when several agents hand work off to each other — a planner, a retriever, a writer — and one frozen golden set is expected to grade the whole chain at once, even though 'correct' means something different at every handoff.

Extend your knowledge

• ▹Open your own eval set right now. Does a single example have a date anywhere on it? If not, that's exactly the gap this lesson is about.
• ▹Pull recent support tickets or complaints tagged 'wrong answer' and cross-check them against your golden set. Is any of that failure pattern sitting in your eval marked as a correct example?
• ▹Running an LLM-as-judge setup? Check whether the judge's rubric has a version and a review date. The rubric goes stale exactly the same way the labels do.
• ▹Go read up on 'data drift' and 'concept drift' from classic ML monitoring literature. This whole problem maps onto both: concept drift when the correct answer itself changes (the refund-policy example), data drift when the input distribution shifts (the edge case that turned common) — just happening to your eval harness instead of your training data.