Fine-tuning Basics: When and Why to Actually Retrain a Model

What Fine-tuning Actually Is

Every large language model starts from pre-training: a base model learns general language patterns from a massive, broad dataset. Fine-tuning takes that already-trained model and continues training it — using a smaller, specific dataset built for one task — which adjusts its weights (the numeric parameters that determine its behavior) further in that direction.

This is a fundamentally different mechanism from prompt engineering or RAG. Both of those change what the model reads in its context window at the moment of a single request — instructions, examples, retrieved documents — but the model’s weights never move. Stop supplying that context, and the model instantly reverts to exactly how it behaved before. Fine-tuning is the opposite: the weight update persists, so the model behaves differently on every future request, with or without any special prompting.

When Fine-tuning Is Actually the Right Call

Fine-tuning is easy to reach for as a vague upgrade — “let’s fine-tune it to make it better.” That instinct is almost always wrong. Fine-tuning is worth its setup cost for a specific shape of problem: a narrow task, that’s stable over time, run at high volume, where prompting has genuinely been tried and keeps failing in a consistent, specific way.

Each of those words is doing real work. Narrow means one well-defined job — not “general intelligence.” Stable means the underlying facts or rules don’t change week to week — fine-tuning bakes a snapshot into the weights, so fast-changing information is a job for retrieval, not retraining. High volume means the setup cost (collecting data, training, evaluating) pays for itself over many requests, not one. And “prompting keeps failing” means you have evidence, not a hunch — a stronger base model or a better prompt might close the gap for free.

Try a few scenarios below and see which of these conditions each one actually meets:

Full Fine-tuning vs. Instruction Tuning

“Fine-tuning” isn’t one single technique — it’s a family of approaches that differ in what gets updated and toward what goal.

Full fine-tuningupdates all (or nearly all) of a model’s weights, trained on a dataset built for one specific, narrow task — the kind of scenario covered above, like a support-ticket classifier or a strict output-format corrector. It’s the most direct approach, and also the most resource-intensive, since every parameter in the model is a candidate for updating.

Instruction tuning is a broader, different goal: training a base model on a large, diverse set of instruction/response pairs so it learns to follow instructions and hold a conversation in general, rather than perform one narrow task. This is actually the step that turns a raw, next-token-predicting base model into the kind of general-purpose assistant most people interact with — it’s a foundational stage most production models already went through, not something most teams do themselves on top of an already-instruction-tuned model.

The practical distinction: instruction tuning aims at general instruction-following ability; full fine-tuning (what most teams actually mean when they say “we fine-tuned a model”) aims at one specific, narrow task on top of a model that’s already instruction-tuned.

The Real Cost and Data Requirements

Concrete dollar figures for fine-tuning vary enormously by model size, provider, and training duration, so a single number here would be misleading. What’s more useful — and verifiable — is the structural shape of the cost, and real numbers on the data side.

On data volume, OpenAI’s own fine-tuning documentation is concrete: the platform requires a minimum of 10 training examples, but recommends starting with around 50 well-crafted examples and evaluating whether that measurably improves results before investing further — improvements are typically seen in the 50–100 example range, with real production use cases often running into the thousands. Critically, the same documentation states the evaluation should come first: build evals, confirm prompting has actually plateaued, and only then invest in preparing a training set.

Source: OpenAI, “Supervised fine-tuning” guide

Structurally, fine-tuning costs more than prompting or RAG along three separate axes: preparing a labeled dataset (real human or verified effort, not automatic), the training run itself (compute cost proportional to model size and dataset size, paid whether or not the result turns out useful), and ongoing hosting of a custom model artifact instead of calling a shared, already-hosted one. Prompting and RAG carry none of that fixed up-front cost — which is exactly why the decision in the section above (a narrow, stable, high-volume task where prompting has plateaued) matters: it’s the condition under which that fixed cost is actually worth paying.

Fun Fact

Most people interacting with a “fine-tuned model” every day don’t realize it — the general-purpose assistant behavior of a chat model (following instructions, holding a conversation, refusing unsafe requests) is itself the product of instruction tuning and related fine-tuning stages applied on top of a raw base model that, left alone, would just predict plausible next tokens without any of that structure.

Test Yourself

Ready for the failure modes that actually show up when fine-tuning goes into production — catastrophic forgetting, overfitting on a narrow dataset, a real training-data format, and how to tell whether a fine-tune actually helped? Continue to the Advanced level →