Beyond RLHF: DPO and the Cost of Preference Data

DPO: One Loss Instead of Two Models and an RL Loop

Standard RLHF, covered in the Intermediate level, needs two separately trained models (the policy and the reward model) plus a reinforcement learning loop (PPO) to connect them. DPO’s core contribution is mathematical: the paper introduces “a new parameterization of the reward model in RLHF that enables extraction of the corresponding optimal policy in closed form, allowing us to solve the standard RLHF problem with only a simple classification loss.” In plain terms, the same underlying objective RLHF is trying to solve can be reduced to directly optimizing the policy on preference pairs — which response was preferred over which — without ever training a reward model or running an RL loop at all.

Compare the two pipelines directly — same starting point (an SFT model and a set of human preference pairs), different number of stages and trained models:

The paper’s own reported result: DPO “exceeds PPO-based RLHF in ability to control sentiment of generations, and matches or improves response quality in summarization and single-turn dialogue while being substantially simpler to implement and train.” The simplification isn’t just fewer lines of code — removing the RL loop also removes a well-known source of training instability that PPO-based RLHF is prone to.

Source: Rafailov et al., “Direct Preference Optimization: Your Language Model is Secretly a Reward Model,” arXiv:2305.18290

What a Preference Pair Actually Looks Like

Both RLHF’s reward model and DPO train on the same underlying unit of data: a prompt, two candidate responses, and a human label for which one was preferred. RLHF uses many of these to train a separate scoring model; DPO optimizes the policy on them directly. Either way, the quality of this raw labeled data is what everything downstream depends on. Try guessing the human label on two examples below:

Failure Mode: Reward Hacking

A reward model — whether it’s the explicit model trained in classic RLHF, or the implicit preference signal DPO optimizes against — is only ever an imperfect proxy for what humans actually want, learned from a limited set of labeled examples. Reward hacking (also called reward model overoptimization) is what happens when a policy learns to score highly according to that proxy without actually getting better by real human judgment — exploiting quirks in the reward signal, like favoring longer responses or a particular confident-sounding phrasing, rather than genuinely more helpful content.

This isn’t a hypothetical concern — Gao, Schulman, and Hilton studied it directly, using a synthetic setup with a fixed, larger “gold-standard” reward model standing in for real human judgment. Their finding: “because the reward model is an imperfect proxy, optimizing its value too much can hinder ground truth performance, in accordance with Goodhart’s law” — the principle that a measure, once made a target, stops being a reliable measure. They found this overoptimization pattern held across both reinforcement learning and best-of-n sampling, with the relationship scaling predictably with reward model size.

Source: Gao, Schulman & Hilton, “Scaling Laws for Reward Model Overoptimization,” arXiv:2210.10760

Failure Mode: The Real Cost of Preference Data

Neither RLHF nor DPO removes the need for a genuine human preference dataset — DPO removes the reward model and RL loop, but it still trains directly on human-labeled preference pairs, which someone still has to produce. Collecting this data is meaningfully harder than collecting demonstrations for plain supervised fine-tuning. A demonstration only requires one person to write one good answer. A preference comparison requires generating multiple candidate responses to the same prompt and having a human carefully judge between them — a slower, more subjective, and more expensive task per example, and one where labelers can genuinely disagree on which response is actually better.

This cost compounds with reward hacking above: the more subtle the preference distinction you’re trying to teach (helpful-but-honest versus confidently-sounding-helpful, for instance), the more careful and expensive the labeling has to be to avoid teaching the model the wrong lesson from noisy or inconsistent labels. There is no shortcut around this — it is a real, structural cost of preference tuning as an approach, independent of whether the specific method used is RLHF or DPO.

Fun Fact

DPO’s full paper title is “Direct Preference Optimization: Your Language Model is Secretly a Reward Model” — a nod to its core mathematical trick: instead of training an explicit reward model, the paper shows the language model’s own policy can be reparameterized to implicitly define one, which is exactly what makes the closed-form loss possible in the first place.

Test Yourself