// ⚖ Algorithms Atlas

Alignment & Preference Optimization

Making models helpful and harmless: RLHF, DPO, Constitutional AI, GRPO and RL with verifiable rewards.

8 core2 extended2017–2025

TIMELINE

2017 Reward Modelin2022 Reinforcement 2022 Constitutional2023 Direct Prefere2024 SimPO (Simple 2024 Group Relative2024 RL with Verifi2024 KTO (Kahneman–2024 ORPO (Odds Rat2025 DAPO

01Core algorithms

Learn these first. Sorted by year.

Reward Modeling (Preference Learning) Learns a scoring function from pairwise human comparisons. 2017core Reinforcement Learning from Human Feedback (RLHF) Train a reward model on human preferences, then optimize the policy with RL. 2022core Constitutional AI / RLAIF Uses a written constitution so AI critiques itself; AI feedback replaces human labels. 2022core Direct Preference Optimization (DPO) Optimizes preferences directly — no reward model or RL loop. 2023core SimPO (Simple Preference Optimization) Uses length-normalized log-likelihood as an implicit reward, reference-free. 2024core Group Relative Policy Optimization (GRPO) Samples a group of answers per prompt and uses in-group relative scores instead of a critic. 2024core RL with Verifiable Rewards (RLVR) Rewards come from programmatic checks like math answers or unit tests. 2024core DAPO A set of GRPO fixes and an open system for long-CoT RL. 2025core

02Extended

Variants and follow-ups worth knowing once the core is clear.

KTO (Kahneman–Tversky Optimization) Aligns from simple good/bad labels instead of pairs. 2024ext ORPO (Odds Ratio Preference Optimization) Merges SFT and preference optimization in one step without a reference model. 2024ext

03Course units that teach these

U51 Alignment: RLHF, DPO & Constitutional AI Reward models, PPO-style RLHF, the DPO family, RLAIF and reward hacking. 10 h7 algorithms4 papers U52 Reasoning Models: RLVR, GRPO & Test-Time Compute Chain of thought, self-consistency, process rewards, GRPO/DAPO, verifiable rewards and the R1 recipe. 12 h12 algorithms5 papers U93 Frontier Risk, Scalable Oversight & Governance Weak-to-strong generalisation, debate, Constitutional AI, risk frameworks and regulation (EU AI Act, NIST AI RMF). 6 h4 algorithms3 papers U94 AI for Science & Domain Applications Graph neural networks, AlphaFold, AlphaGeometry and automated research — linked with AIMap’s “AI × Disciplines” section. 10 h4 algorithms3 papers U95 Tracking the Frontier: Reading & Reproducing Papers The three-pass reading method, tracking arXiv and conferences, reproduction and writing. 6 h3 algorithms0 papers