Stage 2 · How LLMs Work
Transformers & LLM Principles
Tokens, embeddings, attention and the scaling laws behind today's models.
4 lessons2 ready
- 01 Tokens and Embeddings: How Text Becomes Numbers Tokenisation (BPE), vocabularies and the vector space where meaning lives. Intermediate 15 min ✓
- 02 Transformers and Attention, Explained Self-attention, queries/keys/values, multi-head attention and the full decoder block. Intermediate 25 min ✓
- 03 Scaling Laws and Emergence How loss falls with parameters, data and compute, and what Chinchilla changed. Advanced 15 min ✎ In progress
- 04 Modern LLM Architectures: Decoders, MoE, Long Context Decoder-only models, mixture-of-experts, positional encodings and long-context tricks. Advanced 20 min ✎ In progress
Matching units in the new course
Each unit has goals, algorithms, must-read papers, a project and a self-test.
U40 Tokenization & Embeddings BPE tokenisation, per-token pricing, embeddings and semantic similarity. 6 h6 algorithms2 paperslesson ready U41 Transformers & Attention Self-attention, multi-head attention, positional encoding, residuals and normalisation — build a GPT from scratch. 16 h6 algorithms3 paperslesson ready U42 Modern LLM Architecture RoPE, SwiGLU, RMSNorm, GQA/MLA, FlashAttention, sliding windows and context extension. 12 h12 algorithms5 papers U44 Pretraining, Data & Scaling Laws Pre-training data pipelines, scaling laws, Chinchilla, μP, MTP and training stability. 10 h6 algorithms4 papers