Direct Preference Optimization (DPO), introduced by Rafailov et al. (2023), is an alignment method that fine-tunes language models directly on human preference data using a simple classification loss. DPO eliminates the need for an explicit reward model and reinforcement learning, instead deriving a closed-form mapping from the Bradley-Terry preference model to the optimal policy.

Standard RLHF involves a complex pipeline: (1) train a reward model on preference pairs, then (2) optimize the policy via PPO against that reward model while constraining KL divergence from a reference policy. This process is computationally expensive, unstable, and requires careful hyperparameter tuning. DPO asks: can we bypass the reward model entirely and optimize directly from preferences?

The standard RLHF objective maximizes expected reward with a KL penalty:

$$J_{RL}(\pi_\theta) = \mathbb{E}_{(x,y) \sim \pi_\theta} [r(x,y)] - \beta \, \text{KL}[\pi_\theta \| \pi_{\text{ref}}]$$

where $r(x,y)$ is the reward, $\pi_{\text{ref}}$ is the reference (pre-trained) policy, and $\beta$ controls the KL constraint strength.

The optimal policy for this objective has a closed-form solution:

$$\pi^*(y|x) = \frac{1}{Z(x)} \pi_{\text{ref}}(y|x) \exp\left(\frac{1}{\beta} r(x,y)\right)$$

where $Z(x) = \sum_y \pi_{\text{ref}}(y|x) \exp\left(\frac{1}{\beta} r(x,y)\right)$ is the partition function.

Solving for the reward in terms of the policy yields:

$$r(x,y) = \beta \log \frac{\pi^*(y|x)}{\pi_{\text{ref}}(y|x)} + \beta \log Z(x)$$

This shows that the policy implicitly defines a reward function. Substituting into the Bradley-Terry preference model $p(y_w \succ y_l | x) = \sigma(r(x,y_w) - r(x,y_l))$, the partition function $Z(x)$ cancels, yielding the DPO loss.

$$\mathcal{L}_{\text{DPO}}(\pi_\theta; \pi_{\text{ref}}) = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)} \right) \right]$$

where $\sigma$ is the sigmoid function, $y_w$ is the preferred response, $y_l$ is the rejected response, and $\mathcal{D}$ is the preference dataset.

This is a binary cross-entropy loss — the model learns to assign higher implicit reward to preferred responses relative to rejected ones. No reward model, no RL sampling, no PPO.

import torch
import torch.nn.functional as F
 
def dpo_loss(policy_logps_w, policy_logps_l,
             ref_logps_w, ref_logps_l, beta=0.1):
    """DPO loss: binary cross-entropy on log-probability ratios."""
    log_ratios_w = policy_logps_w - ref_logps_w
    log_ratios_l = policy_logps_l - ref_logps_l
    logits = beta * (log_ratios_w - log_ratios_l)
    return -F.logsigmoid(logits).mean()

The DPO gradient increases the likelihood of preferred responses and decreases the likelihood of rejected ones, weighted by how wrong the current model is. When the implicit reward margin is already correct, the gradient is small. When the model assigns too-high probability to rejected responses, the gradient is large. The $\beta$ parameter controls how far the policy can deviate from the reference — higher $\beta$ means a stronger anchor to $\pi_{\text{ref}}$.

• Matches or exceeds PPO-RLHF on sentiment control, summarization, and single-turn dialogue
• Better reward-KL tradeoff frontier than PPO
• Tested on models up to 6B parameters
• Significantly simpler to implement and tune
• Has become the dominant alignment method in practice due to simplicity and effectiveness

• Rafailov et al. "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" (2023)
• Ziegler et al. "Fine-Tuning Language Models from Human Preferences" (2019)
• Stiennon et al. "Learning to Summarize with Human Feedback" (2020)

• Constitutional AI
• Reward Overoptimization