// Algorithm learning path
Post-training & Reasoning Models
From instruction tuning and LoRA to RLHF and DPO, then GRPO, RLVR and DeepSeek-R1 — how ChatGPT and o1/R1 are made.
Level
4 steps19 algorithms4 course units
01Steps
Each step builds on the one before. Open any card for the paper and resources.
Teaching the model to follow instructions
Instruction Tuning (FLAN) Fine-tuning on many instruction–response tasks for zero-shot generalization. 2021core Self-Instruct The model generates its own instruction data for fine-tuning. 2022core LoRA (Low-Rank Adaptation) Freezes the model and learns weight updates as small low-rank matrices. 2021core QLoRA Fine-tunes LoRA on a 4-bit quantized base, fitting big models on one GPU. 2023coreReinforcement-learning foundations
Policy Gradient (REINFORCE) Directly increases the probability of actions that led to high return. 1992core Proximal Policy Optimization (PPO) A stable, simple policy-gradient method that clips update size. 2017core Reward Modeling (Preference Learning) Learns a scoring function from pairwise human comparisons. 2017corePreference alignment
Reinforcement Learning from Human Feedback (RLHF) Train a reward model on human preferences, then optimize the policy with RL. 2022core Direct Preference Optimization (DPO) Optimizes preferences directly — no reward model or RL loop. 2023core SimPO (Simple Preference Optimization) Uses length-normalized log-likelihood as an implicit reward, reference-free. 2024core Constitutional AI / RLAIF Uses a written constitution so AI critiques itself; AI feedback replaces human labels. 2022coreReasoning
Chain-of-Thought Prompting Prompting the model to write intermediate reasoning before answering. 2022core Self-Consistency Samples many reasoning paths and takes the majority answer. 2022core Process Reward Models (PRM) Scores each reasoning step rather than only the final answer. 2023core Group Relative Policy Optimization (GRPO) Samples a group of answers per prompt and uses in-group relative scores instead of a critic. 2024core RL with Verifiable Rewards (RLVR) Rewards come from programmatic checks like math answers or unit tests. 2024core DeepSeek-R1 Reasoning RL Recipe Large-scale outcome-reward RL alone teaches long reasoning and self-reflection. 2025core Test-Time Compute Scaling Spending more inference compute — thinking, sampling, search — for better answers. 2024core DAPO A set of GRPO fixes and an open system for long-CoT RL. 2025core
02Course units for this path
Goals, projects and self-tests that go with these algorithms.
U50 Instruction Tuning & PEFT SFT, synthetic data, LoRA/QLoRA/DoRA, adapters and model merging. 10 h8 algorithms3 paperslesson ready U33 Reinforcement Learning Basics MDPs, Q-learning, DQN, policy gradients, PPO and AlphaZero — the prerequisites for post-training and reasoning models. 14 h12 algorithms4 papers U51 Alignment: RLHF, DPO & Constitutional AI Reward models, PPO-style RLHF, the DPO family, RLAIF and reward hacking. 10 h7 algorithms4 papers U52 Reasoning Models: RLVR, GRPO & Test-Time Compute Chain of thought, self-consistency, process rewards, GRPO/DAPO, verifiable rewards and the R1 recipe. 12 h12 algorithms5 papers
03Other paths
- 1Zero to Deep Learning · From linear regression all the way to ResNet and the precursors of the Transformer — build intuition for how models are trained.24
- 2Understanding an LLM End to End · From tokenisation to the Transformer, modern architectures, pre-training and inference — every part of an LLM, in nanoGPT order.19
- 3AI Application Engineer · For product engineers: retrieval, RAG, agents, inference serving and security — turn AI into reliable applications fast.20
- 4Generative Models & Frontier Research · Diffusion/flow matching, multimodality, new architectures, world models and interpretability — the most active research areas of 2024–2026.26