Retrieval eval sets

Day 25: Build your retrieval eval set from failed searches, not from LLM-written questions

Every decision you made in this series so far (chunk size, embedding model, hybrid weights, reranker) was scored against an eval set. If that eval set flatters your retriever, you spent weeks tuning toward the wrong target, and nothing on your dashboard will tell you.

The hook: two true facts that disagree

Say your retrieval dashboard shows recall@5 of 0.91. Meanwhile the Slack channel next to it keeps filling up with 'couldn't find it, ended up asking Minh' and 'the agent pulled the wrong runbook again.' Both are true. The dashboard measured exactly what it was asked to measure: how well the retriever answers the questions in the eval set. Those just aren't the questions people actually ask. The metric is fine. The exam is the problem.

The mechanism: lexical leakage

The standard advice goes like this: take each chunk, ask an LLM to write a question the chunk answers, and store the (question, chunk) pair as gold. Here's what that produces, set against something a real engineer typed:

GOLD CHUNK (runbooks/ingest-worker.md)
  "The ingest worker applies an exponential backoff policy on HTTP 429
   responses from the vendor API, starting at 2s and capping at 5 minutes.
   Retries are disabled when VENDOR_STRICT_MODE is set."

LLM-GENERATED QUERY
  "What is the exponential backoff policy for HTTP 429 responses
   in the ingest worker?"
   -> shares: exponential, backoff, policy, 429, ingest, worker

REAL QUERY (from a support thread)
  "why does the job keep hammering the API"
   -> shares: API  (and 'job' != 'ingest worker', 'hammering' != 'backoff')

I call this lexical leakage. The LLM had the chunk open in front of it, so it borrowed the chunk's words. Basically, someone wrote the exam from the answer key. Your users don't have the answer key. They describe symptoms ('hammering'). They use local nicknames ('the job'). Often they don't know which concept they're looking for, and that's the whole reason they're searching.

Why leakage hides so well

• ▹Every retriever benefits. BM25 rewards exact term matches, and dense embeddings also tend to score high when the query and chunk share surface vocabulary. Hybrid search and reranking (Days 18 to 22) all look great on a leaked set, because the hard part was done before retrieval even started.
• ▹The gaps between configurations shrink. Once every setup lands somewhere between 0.85 and 0.92, you're picking a winner on noise.
• ▹The queries that matter most never make it in. Generating chunk by chunk rarely gives you vague queries ('the deploy thing is broken'), symptom-first queries that describe the effect and not the cause, wrong-name queries (an old service name, a teammate's nickname for a system), or cross-document queries where the answer is split across two chunks.
• ▹It looks rigorous. A few hundred labelled pairs, a pipeline and a dashboard all read as diligence, so nobody asks where the questions came from.
• ▹Agents make it worse. A coding agent that retrieves context repeats the same failure at every step of a multi-step task, and when the agent reformulates a query, the new wording can drift further from the user's, not closer.

A 10-minute test: measure your overlap gap

Before you rebuild anything, check whether your current golden set leaks. For each pair, compute the fraction of the query's content words that also appear in the gold chunk. Then do the same for 30 to 50 real queries, pulled from search logs or support threads and labelled by hand. Compare the two distributions.

import re, json, statistics

STOP = set("the a an of to in for on and or is are was be how what why "
           "does do i we my our it this that with by from at as can".split())

def toks(s):
    return {t for t in re.findall(r"[a-z0-9_]+", s.lower())
            if t not in STOP and len(t) > 2}

def overlap(query, chunk):
    q = toks(query)
    return len(q & toks(chunk)) / len(q) if q else 0.0

def report(path, label):
    rows = [json.loads(line) for line in open(path)]
    s = [overlap(r["query"], r["gold_chunk"]) for r in rows]
    print(f"{label:10s} n={len(s):4d}  median={statistics.median(s):.2f}  "
          f">=0.5: {sum(x >= 0.5 for x in s) / len(s):.0%}  "
          f"zero: {sum(x == 0 for x in s) / len(s):.0%}")

report("golden_synthetic.jsonl", "synthetic")
report("real_sample.jsonl", "real")

Reading the output: if the synthetic median sits well above the real median, and a noticeable share of your real queries have zero overlap while almost none of the synthetic ones do, your recall number is probably inflated. When we ran it, most synthetic queries shared a majority of their content words with the gold chunk. Plenty of real queries shared one word, or none at all. Our real sample was only a few dozen queries, so I trust the direction and the rough size of the gap, not the decimals. Read yours the same way. This test tells you whether to worry. It won't tell you by how much you're overstating recall.

Recipe part 1: mine failures

Your organisation is already recording people failing to find things. You just haven't been collecting it. Start here:

• ▹Support tickets and internal helpdesk queues. Prioritise the ones closed with a link to a doc that already existed, because that link is very nearly a free gold label.
• ▹Slack/Teams messages matching 'can't find', 'where is', 'anyone know', 'is there a doc for' or 'couldn't find'. The answer is usually somewhere in the thread replies.
• ▹Zero-result and no-click searches from your docs portal or internal search.
• ▹Agent traces where the retrieval tool fired 3+ times with reformulated queries in a single user turn, or where the user said 'no, I meant...'. Keep the user's first wording. Throw away the agent's rewrites.
• ▹Abandoned sessions: someone searches, then asks a human the same question a few minutes later.

The privacy step is not optional. These messages contain colleagues' names, customer identifiers, emails and sometimes secrets. Redact before anything is written to the eval set, keep raw exports out of the repo, and have a person review every entry before it gets committed. Regex covers structured identifiers. For personal names in free text you need NER (e.g. Microsoft Presidio) followed by a human pass.

import re, json

FAIL = re.compile(r"\b(can'?t find|cannot find|couldn'?t find|where (is|are|do)|"
                  r"(does )?anyone know|is there a (doc|page|runbook))\b", re.I)

REDACT = [
    (re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+"), "<EMAIL>"),
    (re.compile(r"<@[A-Z0-9]+>"), "<USER>"),                  # Slack mentions
    (re.compile(r"\b(xox[abp]|ghp|sk)[-_][A-Za-z0-9_-]{10,}"), "<SECRET>"),
    (re.compile(r"\b(cust|acct|org)[-_]?\d{4,}\b", re.I), "<CUSTOMER_ID>"),
    (re.compile(r"\+?\d[\d\s-]{8,}\d"), "<PHONE>"),
]

def redact(text):
    for pat, repl in REDACT:
        text = pat.sub(repl, text)
    return text  # then: NER pass for names, then human review

def mine(slack_export_path):
    for msg in json.load(open(slack_export_path)):
        text = msg.get("text", "")
        if FAIL.search(text) and msg.get("reply_count", 0) > 0:
            yield {
                "query": redact(text),             # original wording, redacted only
                "source": "slack",
                "thread_ts": msg["ts"],            # for the labeller to find the answer
                "gold_doc": None,                  # filled in during labelling
            }

Recipe part 2: label with the people who found the answer

• ▹The gold label comes from whoever resolved the failure: the person who answered the thread, closed the ticket or eventually dug up the doc. They know which doc actually helped, and that's not the same question as which doc looks relevant.
• ▹Keep the original wording exactly, typos and nicknames included. Nobody gets to 'clean up' a query. The mess is the signal. Redaction is the only edit you allow.
• ▹Label at doc level first, chunk level second. People remember the page, not the paragraph. Map to chunks afterwards so that re-chunking (Day 8) doesn't throw the set away.
• ▹Allow more than one gold doc for cross-document questions, and tag them so you can report on them separately.
• ▹Keep the 'no good answer exists' cases as negatives. They show whether your system, and the agent sitting on top of it, can say 'I don't know' instead of confidently returning something nearby. They also give you a list of docs nobody has written yet.

Recipe part 3: where LLMs still help

The rule is simple. An LLM can transform or organise real queries, but it never writes questions from the documents. If the model doesn't see the gold chunk while it generates, it can't leak the chunk's vocabulary.

• ▹Paraphrase real queries into variants. 'Why does the job keep hammering the API' turns into 'ingest keeps retrying nonstop' and so on. The set gets bigger and stays symptom-first. Don't show the model the gold chunk.
• ▹Cluster failures by type (vague, symptom-first, wrong-name, cross-doc, no-answer), so you can report recall per type instead of one blended number.
• ▹Draft relevance labels for humans to check. A model suggesting 'probably runbooks/ingest-worker.md' saves labellers time, but a person confirms every one.
• ▹Don't let it write questions from chunks, 'improve' how a query is worded, or act as the final relevance judge without spot checks.

The reversal: what changed when we re-ran

We re-ran our retriever configurations on the failure-mined set, and recall dropped well below the dashboard figure. The drop wasn't the most useful part, though. The ranking of configurations changed. A larger embedding model that had narrowly won on the synthetic set stopped standing out. On real queries, the gap between dense models was smaller than the gap between different ways of handling the query.

What helped most was on the query side: expanding queries with synonyms and symptom-to-cause mappings ('hammering' → retry, backoff, 429, rate limit; old service names → current names), built straight from the failure set. Looking back, that makes sense. The failures were vocabulary mismatches. A better embedding model only partly closes a vocabulary gap, and a 40-line glossary closes it directly. In an agent setup you can put the same mapping in the retrieval tool's description or in a query-rewrite step, so the agent searches with the organisation's vocabulary rather than the user's.

Keeping the set alive

• ▹Your eval set goes stale just like your docs do. Services get renamed, docs move, new kinds of failure show up. A set you froze in Q1 will be flattering you by Q3.
• ▹Make 'I couldn't find X' a one-click report in the docs portal, the chat assistant and the agent UI. Every report goes through the same redaction → review → label pipeline.
• ▹Give one person a 30-minute weekly review of new entries. Retire entries whose gold doc was deleted, and relabel when docs get merged.
• ▹Version the set and report scores per version. When recall moves, you can tell whether the system changed or the exam did.
• ▹Coming up on Day 26: once your eval set reflects reality, the harder question is whether retrieval metrics predict agent task success at all. Good recall@5 doesn't mean the agent finishes the job.

Monday checklist

• ▹1. Run the overlap test on your current golden set against 30 to 50 real queries you collected by hand. Write down the gap, and don't over-read the decimals.
• ▹2. Mine one source this week. Pick the easiest, which is usually Slack 'where is' threads or zero-result search logs. Redact before you save anything.
• ▹3. Get the resolvers to label each query with a gold doc, or with an explicit 'no good answer'. Keep the original wording.
• ▹4. Re-run your current configurations on the new set. Compare the rankings, not just absolute recall, and break the results down by failure type.
• ▹5. Ship the one-click 'couldn't find it' report, and put a weekly 30-minute review in someone's calendar.

Extend your knowledge

• ▹Read the BEIR benchmark paper (Thakur et al., 2021), especially the part on how the way a dataset is built can tilt results toward lexical retrievers like BM25.
• ▹Try Microsoft Presidio to detect and redact PII in mined messages before they reach the eval set. Keep a human review step after it.
• ▹Read Hamel Husain on LLM evals, particularly his habit of looking at real data and doing error analysis before trusting any metric.
• ▹Run a synthetic test set generator (e.g. Ragas test set generation) and your failure-mined set through the overlap test. Note which failure types the generator rarely or never produces.