DPO vs PPO: The Training Alignment Debate

Why Direct Preference Optimization is replacing PPO in most teams — and the edge cases where PPO still wins.

In 2023, DPO felt like a neat theoretical trick. By 2025, it's the default alignment method for most teams. Here's why, and where PPO still wins.

What DPO Actually Does

DPO (Direct Preference Optimization) shows that the optimal RLHF policy can be derived directly from preference data without an explicit reward model. The reward is implicitly parameterized by the ratio of log probabilities between the policy and reference model. Training becomes a binary classification loss on (chosen, rejected) response pairs.

# DPO loss (simplified)
def dpo_loss(policy_chosen_logps, policy_rejected_logps,
            ref_chosen_logps, ref_rejected_logps, beta=0.1):
    pi_ratios = policy_chosen_logps - policy_rejected_logps
    ref_ratios = ref_chosen_logps - ref_rejected_logps
    logits = pi_ratios - ref_ratios
    return -F.logsigmoid(beta * logits).mean()

DPO Advantages

• No reward model: saves training compute, storage, and the RM's failure modes
• No PPO loop: simpler code, fewer hyperparameters, more stable training
• Same data: works on the same (prompt, chosen, rejected) pairs as PPO
• Reproducible: offline training means results don't depend on sampling randomness

PPO Advantages

• Online data collection: policy can generate new (prompt, response) pairs during training, sampling from its own distribution — this matters at the frontier
• Iterative improvement: reward model and policy can be updated in cycles (Constitutional AI, RLHF-V)
• Higher ceiling: with enough compute and data quality, PPO reaches higher peak quality than DPO
• Explicit reward signal: the RM's scores are interpretable for debugging

DPO Failure Modes

DPO is not perfect. Common issues: distribution shift (preference data may not cover the policy's new outputs), length bias (models learn verbose responses score better), and mode collapse on homogeneous datasets. These are fixable with data diversity and iterative online DPO (oRPO, RAFT, online DPO variants).

• DPO paper (Rafailov et al., 2023)
• A General Theoretical Paradigm (Azar et al., 2023)
• Online DPO (Guo et al., 2024)