Fine-tuning in Production: Real Trade-offs and Failure Modes

Catastrophic Forgetting

Catastrophic forgetting isn’t specific to LLMs or even to recent research — it was first documented in neural networks in 1989, when McCloskey and Cohen showed that training a network sequentially on two different tasks caused it to abruptly and severely lose what it had learned on the first task, far worse than the gradual forgetting seen in human learning. That’s where the name comes from: the loss isn’t a gentle decline, it’s abrupt.

Source: McCloskey & Cohen (1989), “Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem”

The same effect has since been confirmed directly in large language models. A 2023 empirical study specifically measured forgetting during continual fine-tuning of LLMs ranging from 1B to 7B parameters, tracking domain knowledge, reasoning ability, and reading comprehension before and after fine-tuning. Forgetting showed up consistently across that range — and counterintuitively, the severity increased with model scale within that range, which the researchers attribute to larger models starting from a stronger baseline, making any degradation more visible.

Source: Luo et al., “An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning,” arXiv:2308.08747

The illustration below is a hand-authored, illustrative timeline — not real benchmark numbers — showing the shape this takes: a narrow fine-tune improves its own target task while unrelated skills the training data never touched quietly degrade alongside it.

Overfitting on a Narrow Fine-tuning Dataset

Catastrophic forgetting is about unrelated skills getting worse. Overfitting is a different failure, on the target task itself: instead of learning the general pattern behind the training examples, the model memorizes specific quirks of that particular, usually small, dataset — exact phrasings, coincidental patterns, quirks of whoever wrote the examples. Training accuracy looks excellent; real, new, production inputs expose that the model never actually generalized.

This risk is highest exactly where fine-tuning is otherwise most tempting: a small, narrow dataset, prepared quickly, covering too few real variations of the task. The practical mitigation is the same one used across machine learning generally — holding out a validation set the model never trains on, and watching for training performance that keeps climbing while validation performance stalls or drops, which is the concrete signal that memorization has overtaken generalization.

A Real Fine-tuning Data Format

Fine-tuning datasets for instruction-following tasks are commonly prepared as JSONL (JSON Lines) — one complete, valid JSON object per line, each representing a single training example. This mirrors the structure OpenAI’s own supervised fine-tuning API documents: a messages array per line, with the roles a chat model already uses. Below is an illustrative example of that shape, for a support-ticket tagging task:

{"messages": [{"role": "system", "content": "Classify the support ticket into exactly one category: billing, technical, account, or other."}, {"role": "user", "content": "I was charged twice for my subscription this month."}, {"role": "assistant", "content": "billing"}]}
{"messages": [{"role": "system", "content": "Classify the support ticket into exactly one category: billing, technical, account, or other."}, {"role": "user", "content": "The app crashes every time I try to upload a photo."}, {"role": "assistant", "content": "technical"}]}
{"messages": [{"role": "system", "content": "Classify the support ticket into exactly one category: billing, technical, account, or other."}, {"role": "user", "content": "I can't remember the email address I signed up with."}, {"role": "assistant", "content": "account"}]}

Each line is a self-contained example: the same system instruction, a realistic user message, and the exact correct output the model should learn to produce. Real fine-tuning datasets are simply many more lines of this same shape — OpenAI’s documentation specifies a hard minimum of 10 examples per file, recommends starting around 50, and notes real production fine-tunes commonly run into the thousands, which is covered in more depth in the Intermediate level.

Source: OpenAI, “Supervised fine-tuning” guide

When Fine-tuning Is Over-Engineering vs. the Only Real Option

Everything above has a real cost — data preparation, the training run, ongoing hosting of a custom artifact, and now two extra failure modes to guard against. That cost means fine-tuning is over-engineering whenever a cheaper approach was never actually ruled out: a task that isn’t genuinely narrow and stable, a prompt that was never seriously iterated on, or facts that change often enough that retrieval was always the right tool.

It becomes the only real option when the target behavior can’t be described well enough in a prompt to get consistent results — a very specific tone or format that keeps slipping despite explicit instructions, or domain-specific judgment (like the legal jargon example from the Intermediate level) that generalizes better from many labeled examples than from a written description of the rule. The dividing line is evidence, not intuition: has prompting genuinely plateaued on real production examples, or has it just not been tried hard enough yet?

Evaluating Whether a Fine-tune Actually Improved Things

A fine-tune is not self-evidently an improvement just because it completed successfully or because training loss went down — the two failure modes above make that especially true. The evaluation discipline needed here is exactly the one already covered for comparing any two models in Comparing Ollama Models Advanced: build a task-specific evaluation set from real examples, score the fine-tuned model against the original base model on that identical set, and scale that scoring with LLM-as-judge where manual review doesn’t reach far enough.

One addition matters specifically for fine-tuning: the evaluation set has to include examples outsidethe fine-tuning task, not just examples of it. Scoring only the target task would completely miss catastrophic forgetting, since that’s a regression on tasks the eval set never touches if it only tests what the model was tuned to do. The same shadow testing approach — running the fine-tuned model on real traffic, logged but not yet shown to users — applies directly here too, before fully switching over.

Fun Fact

The word “catastrophic” in catastrophic forgetting isn’t just dramatic framing — McCloskey and Cohen chose it specifically because the neural networks they studied in 1989 forgot far more abruptly and severely than the gradual, partial forgetting seen in human memory when people learn something new that resembles something they already knew.

Test Yourself