Fine-Tuning Sounds Impressive in a Board Update — That's Exactly the Problem

Day 15: Fine-Tuning Is Not Your First Move

Someone in the room is about to say "let's fine-tune it" like it's a quick lever to pull. It's not. It's the most expensive fix on the table, and more often than not, a sharper prompt would've closed the gap by end of day.

The three-week detour

I watched a team burn three weeks fine-tuning a model just to get consistent JSON with a specific tone out of a support-ticket classifier. Data collection, labeling, training runs, an eval harness — three weeks, two engineers, gone. What actually shipped it was a rewritten system prompt with three few-shot examples and a stricter output schema, built in an afternoon. Nobody wanted to say the three weeks were wasted, so the fine-tuned model shipped anyway, sitting right next to the prompt fix that made it pointless. That's the sunk-cost tell: once the training run is paid for, the model stays in production even after you've proven you never needed it.

The tell: status move vs engineering decision

Pay attention to how fine-tuning gets pitched. "We trained our own model" plays well in a board update. "We fixed the prompt and added a retrieval step" does not — and that asymmetry does real damage. Teams reach for fine-tuning because it makes a good story, not because it's the cheapest way to close the gap. If fine-tuning gets proposed before anyone has even tried rewriting the prompt, you're not looking at an engineering decision. You're looking at a status move dressed up as one.

The cost nobody puts in the proposal

Fine-tuning gets pitched as a one-time cost — label some data, train, done. It isn't. You now own a model artifact, forever, and ownership comes with a maintenance bill nobody wrote into the proposal.

• ▹Retraining on every base-model upgrade — the vendor ships a better base model, and your fine-tune doesn't inherit any of those gains for free; you retrain or you fall behind
• ▹Versioning — every fine-tune is a new artifact to track, roll back, and explain when someone asks why the output changed between v3 and v4
• ▹Eval drift — the eval set you tuned against goes stale as real traffic shifts, so the model quietly degrades on cases nobody ever checked it against
• ▹Serving overhead — a custom model usually means dedicated inference capacity instead of a shared, autoscaled endpoint
• ▹Institutional lock-in — the one person who understands the training pipeline leaves, and suddenly nobody wants to touch the model

The order you're supposed to try things in

In the AI-era stack you've got four levers, and they cost wildly different amounts. Cheapest and fastest first, most expensive and slowest last.

A sharper prompt means clearer instructions, a few good examples, an explicit output schema — hours of work, not weeks. Retrieval or context engineering means handing the model the right facts at inference time instead of trying to bake them into its weights — this is the RAG move, and it fixes 'the model doesn't know X' far more reliably than fine-tuning ever will, because you update the source data instead of retraining anything. Tool use means letting the model call a function, a calculator, a database query, or another agent instead of forcing it to memorize the answer. Only once all three are exhausted — and you can actually show they were exhausted, not just skipped — does fine-tuning earn a spot in the plan.

The one honest test

Before anyone signs off on a fine-tuning project, ask two questions. First: can you name the failure mode in one sentence? Not "the model isn't good enough" — that tells you nothing. "It uses the wrong tone in escalation replies" or "it hallucinates field names that don't exist in our schema" — that you can act on. Second: did you actually try fixing that named failure with a better prompt, and did it fail, with the attempt on record? If you can't answer both, you're not diagnosing a model problem. You're skipping the cheap experiments because the expensive one sounds better on a roadmap slide. Fine-tuning without this test is a solution looking for a problem.

What fine-tuning is actually for

Fine-tuning doesn't teach a model facts. That's retrieval's job, and it does it better, since you can update facts without a training run. What fine-tuning actually does is narrow the model's output distribution — pin it down to one style, one format, one behavior pattern, at a consistency prompting can't fully guarantee at scale. That's a real use case. It's just a narrow one: not "make the model smarter," but "make the model stop wandering." Tomorrow we go deeper into that distinction, because knowing what fine-tuning is for is the other half of knowing when to reach for it.

Extend your knowledge

• ▹Read Anthropic's prompt engineering docs and see how far a sharper prompt and a few good examples get you before training even enters the conversation
• ▹Look at how RAG gets framed in practice — it competes directly with fine-tuning for 'the model doesn't know X' problems, and it's far cheaper to keep updated
• ▹Take your team's last fine-tuning proposal — or the next one that comes up — and run it against the decision order in this lesson: was prompting actually tried and documented as failed first?
• ▹Tomorrow's Day 16 goes deeper on output distribution narrowing — what fine-tuning is for, once you've actually earned the right to use it