trendingAI & ML — INTERMEDIATE
RLHF and Preference Tuning: Teaching a Model What Humans Actually Prefer
This topic picks up right after the general idea of fine-tuning, covered in Fine-Tuning Basics, and parameter-efficient methods like LoRA, covered in LoRA & PEFT. Both of those cover how a model gets fine-tuned. This page is about a different question: fine-tuned to imitate a demonstration is not the same as fine-tuned to match what humans actually prefer — and RLHF is the training process that closes that gap.
The Quick Answer
Reinforcement Learning from Human Feedback (RLHF)is a training process that optimizes a model directly for “which response do humans prefer,” rather than just having it imitate a fixed set of example answers. Ordinary supervised fine-tuning (SFT) can teach a model to follow a format or imitate a demonstration, but it has no way to express how much betterone acceptable answer is than another — every training example just says “produce this exact output.” RLHF adds two more stages on top of SFT: training a reward model on human preference rankings, then using reinforcement learning to optimize the policy against that reward model. The reward model is what makes this practical — it turns a limited set of human judgments into an automatable, scalable scoring function.
Why Supervised Fine-Tuning Alone Falls Short
Supervised fine-tuning works by showing the model example input/output pairs and training it to reproduce outputs like the ones it was shown — the general mechanism covered in Fine-Tuning Basics. That works well for teaching a model a format, a tone, or a domain. But it has a structural limit: each training example only ever says “this is acorrect answer,” never “this answer is better than that other one, by this much.” A model can get very good at imitating demonstrations while still producing responses that are technically on-topic but unhelpful, evasive, or subtly worse than another response it could have given — because nothing in SFT’s training signal ever compared two of its own outputs against each other.
OpenAI’s InstructGPT paper made exactly this point as its central motivation: scaling up a model’s size doesn’t inherently make it better at following what a user actually wants, and supervised fine-tuning on its own can still leave a model producing outputs that are “untruthful, toxic, or simply not helpful.” Getting a model to genuinely match human preference needed a training signal built from direct comparisons between responses — which is what the next two stages provide.
The Real 3-Stage RLHF Pipeline
The InstructGPT paper is the standard reference for how RLHF is actually done in practice, and it describes the process as three distinct stages, each one building on the last:
1. Supervised fine-tuning (SFT)
Human labelers write demonstrations of the desired behavior for a set of prompts, and the base model is fine-tuned on these demonstrations with ordinary supervised learning. This produces a model that can follow the right general shape of an answer, but — as covered above — has no signal yet about relative preference between different acceptable answers.
2. Reward model training
The SFT model generates several different responses to the same prompt, and human labelers rank them from best to worst. A separate model — the reward model — is trained on these rankings to predict a scalar score for any prompt/response pair: how much a human would tend to prefer that response.
3. RL optimization against the reward model
The SFT model is further fine-tuned using reinforcement learning — specifically Proximal Policy Optimization (PPO) — where the reward model’s score is the signal being optimized. The policy generates a response, the reward model scores it, and PPO updates the policy to produce higher-scoring responses over many iterations.
Try stepping through each stage below to see its concrete input, what actually happens during it, and what it hands off to the next stage:
Why a Separate Reward Model at All?
It might seem simpler to skip stage 2 entirely — why not just have a human rate every single response the model produces during reinforcement learning, directly? The answer is scale: reinforcement learning needs to score a very large number of generated responses over the course of training, far more than could realistically be rated by a live human for every single one. A human sitting in the loop for every RL step would make the whole process impossibly slow and expensive.
The reward model solves this by moving the human effort to one place: a bounded set of preference rankings, collected once. Once trained, the reward model can score any new response the policy generates automatically, instantly, and consistently — turning a fundamentally human, slow judgment into a fast, repeatable, automatable signal that reinforcement learning can run against as many times as it needs to.
Fun Fact
The InstructGPT paper found that its RLHF-tuned model, at 1.3 billion parameters, was preferred by human evaluators over the much larger 175-billion-parameter GPT-3 base model — despite having more than 100x fewer parameters. Preference-tuning what a model optimizes for made a bigger difference to human judgment than sheer model size.
Source: Ouyang et al., arXiv:2203.02155
Test Yourself
Why isn’t supervised fine-tuning (SFT) alone enough to make a model match what humans actually prefer?
In the InstructGPT paper’s 3-stage pipeline, what is trained in stage 2?
Why does RLHF train a separate reward model instead of skipping straight from demonstrations to reinforcement learning?
Ready for a real, verified alternative to full RLHF that skips the reward model and RL loop entirely, and the documented failure modes of preference tuning? Continue to the Advanced level →