// ⚖ Alignment & Preference Optimization

Reinforcement Learning from Human Feedback (RLHF)

Train a reward model on human preferences, then optimize the policy with RL.

人类反馈强化学习 RLHF 2022 Intermediate core

IN PLAIN WORDS

Three stages: supervised fine-tuning, a reward model trained on human rankings, then PPO to maximize reward with a KL penalty. It made InstructGPT/ChatGPT follow instructions.

Used in: ChatGPT, Claude, Llama 2-Chat

TIMELINE · prerequisites → this → what builds on it

2017 Proximal Poli…2018 AI Safety via…2021 Instruction T…2022 Reinforcement Lea…2022 Constitutiona…2022 Automated Red…2023 Direct Prefer…2023 GCG Universal…

01Original paper

Training language models to follow instructions with human feedback

Long Ouyang, Jeff Wu, Xu Jiang, et al. · 2022

arXiv ↗ ✓ Verified · arXiv API

02Where it sits

Learn the left column first; the right column builds on this one.

Learn first

Reinforcement Learning from Human Feedback (RLHF)

Then you can learn · 9

Show 1 more

Sleeper Agents (Backdoors that Persist)

Left: prerequisites · right: builds on this · dashed: related. Full map →

03Learning resources

Official docs, explainers, courses and reference code. All links checked.

04Where the course teaches it

U51 Alignment: RLHF, DPO & Constitutional AI Reward models, PPO-style RLHF, the DPO family, RLAIF and reward hacking. 10 h7 algorithms4 papers

On learning paths:Post-training & Reasoning Models

For AI / data

Verification: verified, checked 2026-10-21.
JSON: /api/learn/algorithms.json (id: rlhf)

01Original paper

02Where it sits

Learn first

Then you can learn · 9

03Learning resources

Explainers

Books

Official docs

Reference code

04Where the course teaches it