DPO

Post-training & alignment · 2023

The idea

RLHF fits a reward model, then runs reinforcement learning against it - two stages, unstable, expensive to tune. DPO shows the optimal RLHF policy has a closed form: skip the reward model, train directly on which of two responses a human preferred.

Verdict

What it fixedReparameterize RLHF's reward model away - preference pairs train the policy directly with one classification loss, no reward model, no PPO rollout.

What it still costsStill needs a fixed dataset of human preference pairs; it optimizes the same objective RLHF does, just without the RL machinery.

Where it sits

Worth having first: Instruction tuning & RLHF

Unlocks: ORPO