Fine-tuning vs prompting vs RAG

Day 26: Fine-tuning vs Prompting vs RAG: Measure Before You Choose

Your prototype is wrong about one time in five, and someone on the team already wants to fine-tune. Before you spend three weeks on that, spend two days finding out what kind of wrong it is. Fine-tuning, prompting and RAG each fix a different kind of failure. If you pick one by gut feel, you can easily spend a month fixing the wrong thing. Today you'll learn how to diagnose the failure first, using about 50 labelled questions.

The standup argument

You've probably sat through this standup. The prototype is wrong roughly one time in five. One engineer wants to fine-tune because the model 'doesn't get our domain'. Another wants a bigger RAG stack: a reranker, a better embedding model, maybe a graph. A third says the prompt is a mess and needs rewriting. All three are competent and all three have anecdotes. Nobody can settle it.

At this stage most teams don't have a model problem. They have a diagnosis problem, because nobody knows where the failures come from. From the outside, a wrong answer looks the same in three different cases: the retriever never found the right document, the model misread a document it had, or the model got the facts right and put them in the wrong format. Those are three diseases, and each needs its own treatment. Teams that skip the diagnosis switch strategy every sprint. A lot of what they call 'RAG failures' are really evaluation failures.

The reframe: one lever per failure type

• ▹Prompting fixes instructions and reasoning. The model had what it needed and didn't use it well.
• ▹RAG (retrieval) fixes missing or wrong knowledge. The model never saw the facts it needed, or the facts it saw were stale.
• ▹Fine-tuning fixes consistent behaviour and format at volume. The substance is right, but the shape, tone or schema keeps drifting, and you run the task often enough that a reliable, cheap model is worth paying for.
• ▹The rule for today: you can't choose a lever until you know which failure you have. Measure first, then choose.

Step 1: Build a 50-question eval from real questions

We covered eval basics earlier in the series, so I won't go over them again. Today the eval has one job: it has to tell retrieval failures apart from generation failures. To do that, each item needs more than a question and an answer.

• ▹Use real questions. Pull them from production logs, support tickets, the Slack channels where people ask the bot (or a human), and whatever your pilot users actually typed. Questions you write yourself come out clean and well phrased. Real ones have acronyms, typos, and two questions jammed into one, and those are the ones that break systems.
• ▹Aim for about 50. That's enough to show patterns, and one person can still label it in a day or two. Cover your main document types and the questions users ask most often. Include a few where the correct answer is 'I don't know'.
• ▹Record three things for each question: (1) the expected answer, (2) the supporting passage, meaning the exact text span in your corpus that justifies the answer, and (3) a pass/fail rule that a second person would apply the same way you do.
• ▹Store the supporting passage as text, not as a chunk ID. Chunk IDs change as soon as you re-chunk, and you will re-chunk. A check for whether any retrieved chunk contains the support text keeps working after that.
• ▹Have a domain owner sign off on the expected answers. If engineers label policy questions on their own, the eval measures the engineers' guesses.

{"id": "q017",
 "question": "Can I carry over unused annual leave into next year?",
 "source": "helpdesk ticket, 2026-08",
 "expected_answer": "Up to 5 days, only with manager approval before 1 Dec.",
 "support_text": "Unused leave lapses at year end, except that up to five days may be carried over where approved by the employee's manager before 1 December.",
 "pass_rule": "Must state the 5-day cap AND the manager-approval condition. Citing the leave policy is required.",
 "expected_format": "plain answer, max 3 sentences, one citation"}

Step 2: Sort every failure into exactly one bucket

Run the eval. For each question, log the retrieved chunks, the final answer and whether it passed. Then put every failure into one of three buckets. Check the buckets in the order below, because a miss early in the pipeline hides anything that went wrong after it.

• ▹(a) Retrieval miss: the supporting passage was not in the context the model saw. No prompt can fix this, because the model never had the facts.
• ▹(b) Generation miss: the supporting passage was in the context, but the answer was still wrong in substance. The model misread the passage, confused it with another chunk, hedged, or ignored it.
• ▹(c) Behaviour miss: the substance was right, but the format, tone, length, citation style or schema was wrong.

def retrieved_support(case, retrieved_chunks):
    # text match, not chunk-ID match, so it survives re-chunking
    needle = normalise(case["support_text"])
    return any(needle in normalise(c.text) for c in retrieved_chunks)

def bucket(case, run):
    if run.passed:
        return "pass"
    if not retrieved_support(case, run.retrieved_chunks):
        return "retrieval_miss"
    if not run.substance_correct:      # human label or rubric-based LLM judge
        return "generation_miss"
    return "behaviour_miss"           # right facts, wrong shape

Two practical notes. First, long support passages rarely match exactly after a re-chunk, so in practice you'll match on one key sentence or use fuzzy overlap instead of an exact substring. Second, flag any answer that passes even though the support text wasn't retrieved. In that case the model is answering from its training data, and that only works until the policy changes.

This is what a sorted failure log looks like for an internal policy assistant:

| #  | Question                                          | What happened                                                                 | Bucket     |
|----|---------------------------------------------------|-------------------------------------------------------------------------------|------------|
| 1  | Can I carry over unused annual leave?             | Said 'no'. The exception clause sat in the next chunk and wasn't retrieved    | Retrieval  |
| 2  | Dinner reimbursement cap in Singapore?            | Gave the global cap. Country table was split from its header row              | Retrieval  |
| 3  | Who approves contractor access to prod?           | Retrieved the superseded v2 policy instead of v3                              | Retrieval  |
| 4  | What does EOR mean in the contractor policy?      | Acronym query matched nothing useful in embedding search                      | Retrieval  |
| 5  | Does travel policy cover premium economy?         | Right passage in context; said 'yes', but it only applies to flights > 6h     | Generation |
| 6  | Can I expense a home monitor?                     | Right passage in context; answered 'it depends' without the actual rule       | Generation |
| 7  | Sick days without a medical certificate?          | Right passage among 12 chunks; used the number from another country's section | Generation |
| 8  | Summarise parental leave in 3 bullets             | Correct content, 7 bullets                                                    | Behaviour  |
| 9  | Extract approval fields from this request (JSON)  | Correct values, keys renamed, 'approver_role' missing                         | Behaviour  |
| 10 | Customer-facing: notice period for cancellation?  | Correct, but no citation and too casual for a client channel                  | Behaviour  |

Counts: Retrieval 4 · Generation 3 · Behaviour 3

Step 3: Map each bucket to a lever, cheapest first

Once you have the counts, start on the largest bucket and try its levers in order of cost, cheapest first. After every change, re-run the whole eval, not only the questions you were targeting. A fix in one bucket often breaks something in another.

| Bucket           | Try first (cheap, hours)                                                                                                                                  | Try next (more cost)                                                       | Fine-tune?                      |
|------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------|---------------------------------|
| Retrieval miss   | Chunk by structure (sections, clauses) so rules stay with their exceptions; prefix chunks with doc + section title; metadata filters (version, region, effective date); query rewriting / acronym expansion; add keyword (BM25) search alongside vectors | Swap embedding model; add a reranker; tune top-k                           | No. It's an unreliable way to add facts and can't keep up with document changes |
| Generation miss  | Restructure the prompt (instructions first, context clearly delimited); require citations to passage IDs; allow 'not in the documents'; send fewer, tighter chunks | Stronger model; explicit reasoning step; split into extract-then-answer    | Rarely                          |
| Behaviour miss   | Few-shot examples of the exact format; structured outputs / JSON schema mode; output validator with retry; tone rules in the system prompt                | Smaller task-specific prompts per output type                              | Yes, if it survives all of this and volume is high |

Look at where the expensive options sit in the table. In the retrieval row, a new embedding model and a reranker come after chunking and metadata. If a chunk has already cut the exception off its rule, a better embedding model can't put it back. In the behaviour row, structured outputs come first. Most current model APIs can constrain output to a JSON schema, and that one feature covers much of what teams used to fine-tune for.

From PhoenixDX: the fine-tune we didn't start

We had an internal assistant that answered questions over a client's policy documents, and it wasn't good enough. In planning, the team proposed fine-tuning because the model 'didn't understand their policies'. I asked for a few days to build the eval first. We pulled real questions from tickets and chat threads, had the policy owner write the expected answers, and put every failure in a bucket.

Most of the failures were retrieval misses, and most of those followed one pattern. Our fixed-size chunker kept separating a policy statement from the 'except where...' paragraph that followed it. The model received the general rule and stated it confidently, without the exception. We re-chunked on section boundaries, kept each exception attached to its parent clause, and prefixed every chunk with its document and section title. Then we made one prompt change: cite a passage for every claim, and say 'not found in the documents' when no passage supports an answer. After that, the retrieval bucket shrank sharply.

Some failures stayed. A few generation misses on numeric tables wouldn't go away because the model kept reading values from the wrong row, and we ended up treating tables as a separate problem. One behaviour miss, an output format for a downstream system, we fixed with a schema validator and retry instead of a fine-tune. The eval also took longer than we planned, because the authoritative expected answers had to come from someone outside engineering. That's normal, so build it into your plan.

When fine-tuning really is the answer

Fine-tuning has a real place, but it's a narrower one than most people assume. It's worth the cost when all of the following are true:

• ▹The task is narrow and runs at high volume. Examples: classification, extraction to a fixed schema, routing, tool-call formatting, or a house style applied thousands of times a day.
• ▹After you've fixed retrieval and prompting, behaviour misses are the largest bucket left in the eval.
• ▹Prompting, examples and structured outputs have stopped improving results, or they only work with a large, expensive model and long prompts.
• ▹A smaller fine-tuned model would cut cost or latency by an amount that matters at your volume.
• ▹You have enough clean, labelled examples to train on, and your eval set is kept out of the training data. If you train on the eval set, its scores no longer mean anything.

The most important caveat is that a fine-tune is tied to one base model. When the provider releases a new model or retires the old one, you have to re-train and re-evaluate, and sometimes you'll find the new base model handles the task without a fine-tune. So treat the fine-tuned weights as disposable. The eval set is what you keep, and it's what lets you make that decision in an afternoon rather than over a quarter.

The agentic twist: diagnose per role, not per product

Over the next few days we move into multi-agent systems. The same diagnosis works there with one change: you run it separately for each role. In a pipeline with a planner, a retriever and a tool-caller, each role can end up in a different bucket.

• ▹Retriever agent: usually retrieval misses, such as the wrong source, a stale version or a poor query rewrite.
• ▹Planner: often generation misses, like a bad decomposition or skipped steps. It can also have behaviour misses, where the plan isn't in the format the orchestrator parses.
• ▹Tool-caller: mostly behaviour misses, like wrong argument names or invalid schemas. Small fine-tuned models often pay off here because the task is narrow and runs constantly.
• ▹What this means in practice: you need traces that record each role's input and output, plus eval items that check each step and not only the final answer. An end-to-end score tells you that something failed. It doesn't tell you which agent.

Monday checklist

• ▹Collect 50 real user questions. For each one, write the expected answer, the supporting passage text and a pass/fail rule.
• ▹Add a column, 'Supporting passage retrieved? Y/N', filled in automatically by text match.
• ▹Put every failure in a bucket (retrieval, generation or behaviour) and post the counts somewhere the whole team can see them.
• ▹Work on the largest bucket, cheapest levers first. Re-run the full eval after every change.
• ▹Agree a team rule: nobody proposes a fine-tune until the behaviour bucket is the largest one left.

Exercise for Day 26

Run your current prototype against 20 real questions. For each failure, record three things: whether the supporting passage was retrieved, whether the answer was right in substance, and whether the format was right. Post three numbers: retrieval misses, generation misses and behaviour misses. Then write one sentence saying which lever you'd pull first and why. If retrieval is your largest bucket, open the chunks behind three of those misses and look at where the splits fall. The cause is often right there. Tomorrow, on Day 27, we take the per-role idea into agent pipelines and look at how to evaluate a system where any one of several steps could be the one that failed.

Extend your knowledge

• ▹Hamel Husain's blog post 'Your AI Product Needs Evals': a hands-on guide to building evals from real traces and doing error analysis before you pick a fix.
• ▹Anthropic's post 'Introducing Contextual Retrieval': a specific technique for the retrieval bucket. It adds document context to each chunk and combines BM25 with embedding search.
• ▹The Ragas library docs, in particular the context recall and faithfulness metrics. These map roughly onto the retrieval and generation buckets.
• ▹Chip Huyen's book 'AI Engineering' (O'Reilly): see the chapters on evaluation and on finetuning, including when to finetune and when to use RAG.