We Trained a LoRA. It Got Rolled Back Two Weeks Later.

The LoRA That Shouldn't Have Been Trained

Fine-tuning feels like the grown-up move. Versioned, trainable, real ML — the kind of thing you can point to in a design review and say we did the work. Most of the time, though, it's solving a problem that three good examples in a prompt would've fixed for free.

The tell

A team I worked with had an agent that kept misrouting support tickets into the wrong queue. Someone trained a LoRA on a few hundred labeled examples. It shipped, it got a party-emoji in Slack, everyone moved on to the next thing. Two weeks later it got quietly rolled back — not because it broke anything, but because a rewritten system prompt matched its accuracy, and the LoRA had cost something the prompt never did: a training pipeline to maintain, a checkpoint to version, a new step in deploy that somebody now had to own. Task performance barely moved. Everything around the task got heavier.

What a LoRA actually is

LoRA — Low-Rank Adaptation — freezes the base model's weights and bolts on a small pair of trainable low-rank matrices at each targeted layer. Instead of updating billions of parameters, you're training a few million. Cheap to train, cheap to store, cheap to swap in and out at inference by adding the adapter's delta back onto the frozen weights. That's the whole pitch: behavior change without the cost of a full fine-tune. It's a legitimate, useful technique. It is not a substitute for telling the model clearly what you want.

The question nobody asked first

Before anyone touched a training script, someone should have asked: have we actually exhausted context engineering? Nobody did. When I went back and read the original system prompt, here's what was in it:

• ▹No examples — just an abstract instruction ('classify the ticket into the correct queue')
• ▹No explicit failure cases — the prompt never named the common misroutes or explained why they were wrong
• ▹No eval harness — no way to measure 'improvement' beyond vibes and a handful of manually reviewed tickets, so nobody could actually tell if the LoRA helped or the team just believed it did

Watch for this pattern in agentic systems generally: a model 'failing' at a task is often a specification failure wearing a capability-failure costume. The model wasn't incapable. It was never told what the task actually required.

The rewrite

Same base model. No training run. The fix was a rewritten system prompt with three well-chosen examples — one of them the exact case the model kept blowing — plus a short, explicit list of common failure patterns to avoid. Accuracy matched the LoRA's, and on the eval set built to actually check this, it edged slightly ahead. The whole 'fix' was a few hundred tokens anyone on the team could read, argue with, and change in a pull request.

Why the LoRA still 'worked' — and was still wrong

The LoRA wasn't fake progress. It genuinely nudged the model's behavior toward the labeled examples — that's what gradient descent on a few hundred pairs does. But it was solving the wrong layer of the problem. The task didn't need new capability baked into the weights; it needed the existing capability pointed at a clearer target. A LoRA trained to fix an underspecified prompt just encodes that fix into opaque weights instead of a human-readable instruction. Same output, minus anyone's ability to read why.

The actual decision rule

• ▹You have a working eval — a way to measure the task numerically, not by reading transcripts and feeling good about them
• ▹You have a prompt-engineered baseline you've genuinely tried to break — examples, explicit failure cases, structured output constraints, maybe a critic pass — and it's still short
• ▹The failure left is about behavior or style consistency at scale, not missing instructions — the model 'knows' the right answer in isolated cases but drifts under load, long context, or volume where a prompt budget can't cover every case

If you can't check all three boxes, you don't have a fine-tuning problem yet. You have an underspecified prompt with an eval you haven't built.

Where LoRA earns its place

The honest exception: consistent formatting, tone, or voice across thousands of generations, at a cost or latency budget where a long few-shot prompt doesn't fit — a high-volume agent pipeline where every extra token in the system prompt multiplies across millions of calls. There, a LoRA trained on the style you want can outperform a prompt that's fighting for space. That's a real, narrow case. It is not the default case.

The cost nobody counts

A LoRA is a versioned artifact. Someone owns it now — retraining it when the base model updates, explaining why it exists in a design review eighteen months from now, debugging why it interacts oddly with a new prompt someone else stacked on top. A prompt is a line of text anyone on the team can read, question, and edit tomorrow in a five-minute PR. When you're deciding what to reach for, that maintenance asymmetry should weigh as much as the accuracy number.

Extend your knowledge

• ▹Read the original LoRA paper (Hu et al., 2021, 'LoRA: Low-Rank Adaptation of Large Language Models') to understand the rank and alpha hyperparameters you'd actually be tuning.
• ▹Before your next fine-tuning discussion, write down your eval metric and baseline prompt accuracy first — if you can't, you're not ready to have the discussion.
• ▹Look at Anthropic's and OpenAI's prompt engineering guides for few-shot example selection — most 'capability gaps' close there before any training is needed.
• ▹If you've already shipped a LoRA, go back and ask whether a rewritten prompt with the LoRA's training examples folded in as few-shot examples would match it — you may already have the eval data to check.