// ⚖ Alignment & Preference Optimization
Direct Preference Optimization (DPO)
Optimizes preferences directly — no reward model or RL loop.
直接偏好优化 DPO 2023 Intermediate core
IN PLAIN WORDS
RLHF's optimal policy has a closed form, so the preference loss can be written on the policy itself: raise the preferred answer's likelihood relative to a reference model. Simple, stable and widely used.
Used in: Zephyr, Tulu, Llama 3 post-training
TIMELINE · prerequisites → this → what builds on it
01Original paper
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Rafael Rafailov, Archit Sharma, Eric Mitchell, et al. · 2023
arXiv ↗ ✓ Verified · arXiv API
02Where it sits
Learn the left column first; the right column builds on this one.
Left: prerequisites · right: builds on this · dashed: related. Full map →
03Learning resources
Official docs, explainers, courses and reference code. All links checked.
04Where the course teaches it
On learning paths:Post-training & Reasoning Models
For AI / data
- Verification: verified, checked 2026-10-21.
- JSON: /api/learn/algorithms.json (id:
dpo)