// ⚖ Alignment & Preference Optimization

Constitutional AI / RLAIF

Uses a written constitution so AI critiques itself; AI feedback replaces human labels.

宪法式 AI / RLAIF 2022 Intermediate core

IN PLAIN WORDS

The model critiques and revises harmful answers per principles for SFT, then AI-generated preferences drive RL, reducing human labeling of harmful content.

Used in: Anthropic Claude

TIMELINE · prerequisites → this → what builds on it

2022 Reinforcement…2022 Constitutional AI…

01Original paper

Constitutional AI: Harmlessness from AI Feedback

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, et al. · 2022

arXiv ↗ ✓ Verified · arXiv API

02Where it sits

Learn the left column first; the right column builds on this one.

Learn first

Reinforcement Learning from Human Feedback (RLHF)

Constitutional AI / RLAIF

Then you can learn

Nothing in the atlas builds on it yet

Left: prerequisites · right: builds on this · dashed: related. Full map →

03Learning resources

Official docs, explainers, courses and reference code. All links checked.

Explainers

Anthropic ↗EN

Books

Nathan Lambert：RLHF Book ↗EN

04Where the course teaches it

U51 Alignment: RLHF, DPO & Constitutional AI Reward models, PPO-style RLHF, the DPO family, RLAIF and reward hacking. 10 h7 algorithms4 papers U93 Frontier Risk, Scalable Oversight & Governance Weak-to-strong generalisation, debate, Constitutional AI, risk frameworks and regulation (EU AI Act, NIST AI RMF). 6 h4 algorithms3 papers

On learning paths:Post-training & Reasoning Models

For AI / data

Verification: verified, checked 2026-10-21.
JSON: /api/learn/algorithms.json (id: constitutional-ai)