Stage 2 · How LLMs Work
Pre-training, Data, Fine-tuning & Alignment
How base models are built and then turned into helpful, harmless assistants.
5 lessons1 ready
- 01 How LLMs Learn: Pre-training, Fine-tuning and RLHF The three-stage recipe behind ChatGPT-style assistants, and newer methods like DPO. Intermediate 25 min ✓
- 02 Pre-training Data: Collection, Cleaning and Mixing Web crawls, deduplication, quality filters, data mixtures and licensing. Advanced 20 min ✎ In progress
- 03 Fine-tuning in Practice: SFT, LoRA and QLoRA When to fine-tune, how parameter-efficient methods work and how to evaluate the result. Advanced 25 min ✎ In progress
- 04 Alignment Methods: RLHF, DPO and Constitutional AI Reward models, preference optimisation, AI feedback and their failure modes. Advanced 20 min ✎ In progress
- 05 Reasoning Models and Reinforcement Learning Chain-of-thought, RL with verifiable rewards and spending more compute at test time. Expert 20 min ✎ In progress
Matching units in the new course
Each unit has goals, algorithms, must-read papers, a project and a self-test.
U44 Pretraining, Data & Scaling Laws Pre-training data pipelines, scaling laws, Chinchilla, μP, MTP and training stability. 10 h6 algorithms4 papers U50 Instruction Tuning & PEFT SFT, synthetic data, LoRA/QLoRA/DoRA, adapters and model merging. 10 h8 algorithms3 paperslesson ready U51 Alignment: RLHF, DPO & Constitutional AI Reward models, PPO-style RLHF, the DPO family, RLAIF and reward hacking. 10 h7 algorithms4 papers U52 Reasoning Models: RLVR, GRPO & Test-Time Compute Chain of thought, self-consistency, process rewards, GRPO/DAPO, verifiable rewards and the R1 recipe. 12 h12 algorithms5 papers