We Spent Weeks Fine-Tuning Our Coding Agent. A Stock Model Beat It Overnight.

Why this matters

Every few months, a new base model ships — cheaper, sharper, sometimes both. If the value in your AI feature is baked into fine-tuned weights, each of those upgrades is a redo: retrain or fall behind. If that value lives in your context pipeline, the upgrade lands for free. That's a build-vs-buy call worth making on purpose, before a training budget disappears into finding it out the hard way.

The morning a better model beat our own fine-tuned agent

We had a fine-tuned coding agent in production — trained on months of our own codebase patterns, humming along, doing its job. Then a new base model shipped. Out of curiosity, someone pointed the stock, un-tuned version of the same agent at it: no training at all, just a repo map, retrieval over our docs, and a handful of past-failure examples stuffed into the prompt at call time. It beat our fine-tuned agent. Not by a landslide — but enough to sting. The model we'd spent weeks training was now the weaker one, and the 'lazy' version that did its homework at inference time had just inherited a free upgrade.

What we built, and why fine-tuning felt like the 'proper' choice

At the time, the instinct made sense. We had thousands of examples of how our codebase handled errors, naming, module boundaries — patterns that felt too nuanced to spell out in a prompt. Fine-tuning let us bake all of that in once, then treat every inference as 'just the API call': cheap, fast, nothing to assemble per request. It felt like proper engineering — pay the cost upfront, never think about it again. That instinct is exactly the trap.

• ▹Fine-tuning: train the base model on your examples once, and the patterns become part of the weights.
• ▹Inference-time context: assemble the relevant patterns — retrieval, repo maps, failure examples — fresh, on every single call.
• ▹Both are chasing the same outcome: a model that behaves like it knows your codebase. They just store that knowledge in different places.

The hidden bill: pinned to one model version

Fine-tuning doesn't stop costing you after the training run. It pins your investment to one exact base-model checkpoint. When the vendor ships something better — sharper reasoning, longer context, a lower price — your fine-tuned weights don't move an inch. You're left with two bad options: stay on the older, now-inferior base model to protect what you trained, or re-collect examples and retrain against the new one just to get back to where you already were. We hit this wall on every major release. The retraining wasn't making the agent better — it was treading water.

What the context pipeline did instead

The un-tuned agent's 'expensive' part — retrieval over our docs, a generated repo map, a curated set of past failures — got assembled fresh at call time and handed to whatever model sat behind the API. Per call, that cost more than a fine-tuned model's plain prompt. But when the new base model shipped, we changed one model string. Nothing else. The pipeline didn't know or care which model was answering — it just handed over better context, and the new model made better use of it, automatically.

The reframe: frozen artifact vs compounding system

Fine-tuning is an investment in an artifact: a specific set of weights, frozen the moment training finished. Context engineering is an investment in a system: the retrieval index, the repo-map generator, the failure-example store. An artifact starts depreciating the second a better base model exists. A system compounds — every improvement to retrieval quality, every failure example you log, every base-model upgrade, all stack on top of what's already there. It's the difference between buying a car and investing in the road network. Only one of those gets better because someone else improved the cars.

The falsifiable rule

• ▹Fine-tuning wins when the domain is narrow and stable — it won't shift under a new model's different behavior — or when call volume is extreme enough that per-call context tokens dominate your bill, or when latency is tight enough that assembling context at request time simply isn't an option.
• ▹Inference-time context wins when the underlying models are moving fast (in 2026, they are), your call volume is moderate rather than extreme, and you need auditability — the ability to point at the exact doc or example behind an answer, which a baked-in weight will never give you.
• ▹If your team can't say, plainly, which of these two regimes you're in before you start — that's the signal you're not ready to commit to a fine-tune yet.

Three-question checklist before you commit to a fine-tune

• ▹1. Will this domain still look the same in six months, or is it tied to a codebase or product that changes weekly? If it changes weekly, context wins — you'd be retraining on a treadmill anyway.
• ▹2. Is our call volume actually high enough that the extra tokens per call show up as real money — or are we optimizing a bill we don't have yet?
• ▹3. If a meaningfully better base model ships next quarter, what does adopting it cost under each approach — and can we actually stomach that cost under the one we're about to pick?

Extend your knowledge

• ▹Read what Anthropic and OpenAI have each published on fine-tuning versus retrieval-augmented prompting — both labs have written directly about this tradeoff for their own APIs.
• ▹Look at how a coding agent like Aider builds a repo map, and how retrieval tools index a codebase, instead of fine-tuning per repo. It's the production version of the pipeline described here.
• ▹If you already have a fine-tuned model running, run the falsifiable rule against it honestly: which regime are you actually in, and would you make the same call today?
• ▹Next time a new base model ships, measure the delta on your context pipeline before you touch a line of training code. It's the fastest way to find out whether you're compounding or just resetting.