What is Direct Preference Optimization (DPO) and how does it simplify RLHF?

DPO removes the reward model and directly optimizes preferences using a reparameterized loss.

Direct Preference Optimization reformulates RLHF as a purely supervised objective derived from preference pairs. Instead of training a reward model and running policy optimization, DPO directly optimizes the policy using a likelihood ratio between preferred and dispreferred outputs. This stabilizes training by removing high-variance reinforcement steps while still aligning with human preferences.