Skip to content
English

Switch theme

Discover▾

Not sure where to start? This week’s hottest software, models and stacks in three minutes.

AI Apps▾

AI software people use: from assistants like ChatGPT to tools for coding, video and workflows, picked by purpose.

For Agents▾

Software for AI: tools that let your agent (Claude Code, Codex, Cursor…) browse, read docs, run code and remember.

Models▾

Choose models, estimate cost, self-host: prices, rankings, benchmarks, free tiers and compute.

Solutions▾

A goal or an error message? Ready-made stacks, steps and fixes by scenario.

Learn▾

A systematic course from zero to the frontier, with an algorithm library, must-read papers and tutorials.

Frontier▾

Daily AI news, weekly highlights, the latest papers and how AI is changing every discipline.

// Stage 5 · Post-training, Alignment & Reasoning Models

Alignment: RLHF, DPO & Constitutional AI

Reward models, PPO-style RLHF, the DPO family, RLAIF and reward hacking.

What you will learn ↓ Code & exercises ↓

about 10 hours7 algorithms4 paperslesson text in progress

01Goals

Do these first: Instruction Tuning & PEFT · Reinforcement Learning Basics

02Algorithms to master

Each card has the original paper, prerequisites and resources.

Reward Modeling (Preference Learning) Learns a scoring function from pairwise human comparisons. 2017core Reinforcement Learning from Human Feedback (RLHF) Train a reward model on human preferences, then optimize the policy with RL. 2022core Direct Preference Optimization (DPO) Optimizes preferences directly — no reward model or RL loop. 2023core KTO (Kahneman–Tversky Optimization) Aligns from simple good/bad labels instead of pairs. 2024ext ORPO (Odds Ratio Preference Optimization) Merges SFT and preference optimization in one step without a reference model. 2024ext SimPO (Simple Preference Optimization) Uses length-normalized log-likelihood as an implicit reward, reference-free. 2024core Constitutional AI / RLAIF Uses a written constitution so AI critiques itself; AI feedback replaces human labels. 2022core

03Must-read papers

04Courses & videos

05Hands-on: project, code & exercises

Free, official or author-published pages. Most run in the browser or on a free Colab GPU.

PROJECT

Run DPO with TRL on a public preference dataset for a ~1B model and compare helpfulness and safety before and after.

  1. 1
    TRL：DPO Trainer ↗Official docs

    Train a small model with DPO on a preference dataset and compare answers before and after.

  2. 2
    github.com ↗Reference code

    Read Tülu’s full SFT → DPO → RLVR post-training scripts.

06Self-test

Answer in your head, then open to check.

What does the KL penalty do in RLHF?

It stops the policy drifting too far from the reference model and gaming the reward.

Which two things does DPO remove?

A separate reward model and online RL sampling.

In Constitutional AI, what does “AI feedback” replace?

Human preference labels on harmfulness.

On these routes: Algorithm Researchers

Data updated: AI algorithms & curriculum(UTC+8)