What is Reinforcement learning from human feedback?

How it works

The idea of learning a reward from human comparisons was published in 2017 by Christiano and colleagues for control tasks. OpenAI's 2022 InstructGPT paper applied it to LLMs in three steps that became the standard recipe:

1. Supervised fine-tuning. Labellers write good answers to sample prompts, and the pre-trained model is fine-tuned on them.
2. Reward model. The model produces several answers per prompt, labellers rank them, and a separate reward model is trained to predict which answer people prefer.
3. Reinforcement learning. The language model generates answers, the reward model scores them, and an algorithm (PPO in the paper) updates the model to earn higher scores, with a penalty that keeps it from drifting too far from the original.

Newer methods simplify or replace parts of this. Direct Preference Optimization (DPO) learns from preference pairs without a separate reward model, and some labs use AI-generated feedback against written principles. Reinforcement learning with automatically checkable rewards, such as passing tests or correct maths answers, is central to training reasoning models.

Why it matters

• It made chat assistants usable. The InstructGPT paper reported that people preferred outputs from a 1.3B-parameter InstructGPT model over the 175B GPT-3 model, despite it being over 100 times smaller.
• It shapes behaviour, not just knowledge. Helpfulness, tone, refusals and honesty are largely tuned at this stage.
• It has known side effects. Models can learn to please raters rather than be right, which shows up as flattery (sycophancy) or confident-sounding answers. Reward models can also be "gamed".

Example

Given "Explain inflation to a 10-year-old", the model drafts four answers. Raters prefer the short one with a pocket-money example over a textbook definition. Across thousands of such rankings, the reward model learns "clear, pitched at the audience", and RL pushes the model toward that style everywhere.