How does Reinforcement Learning from Human Feedback (RLHF) improve ChatGPT?

RLHF aligns model outputs with human preferences using reward models and reinforcement learning.

RLHF involves three stages: supervised fine-tuning, reward modeling, and policy optimization. Human evaluators rank outputs, and a reward model learns to predict these preferences. The language model is then optimized using reinforcement learning (often PPO) to maximize reward scores.

This process improves helpfulness, safety, and alignment with human intent beyond raw next-token prediction.