dpo-guide

• SKILL.mdcore · 7,014 bytes

## Overview

Direct Preference Optimization (DPO) trains a language model directly on preference
pairs — (prompt, chosen, rejected) — without training a separate reward model or
running reinforcement learning. The key mathematical insight is that the optimal
policy under a KL-constrained reward objective has a closed form, which lets you
reframe "maximize this implicit reward" as a simple classification loss on the
policy itself: increase the likelihood of the chosen response relative to the
rejected one, scaled by how far the policy has moved from a reference model.

In practice, DPO is dramatically simpler to run than PPO-based RLHF: it's a single
supervised-style training loop with standard hyperparameters, stable gradients,
and no value network, advantage estimation, or KL scheduling. It has become the
default preference-tuning method for most practitioners. Its limits: it's an