// Algorithm learning path
Understanding an LLM End to End
From tokenisation to the Transformer, modern architectures, pre-training and inference — every part of an LLM, in nanoGPT order.
Level
4 steps19 algorithms5 course units
01Steps
Each step builds on the one before. Open any card for the paper and resources.
How text becomes numbers
The Transformer core
Transformer & Self-Attention An attention-only, parallelizable sequence model — the backbone of all LLMs. 2017core GPT-3 & In-Context Learning Large enough language models learn new tasks from a few examples in the prompt. 2020core BERT A bidirectional Transformer encoder pretrained by masked-word prediction. 2018core Rotary Position Embedding (RoPE) Encodes position by rotating query/key vectors so attention sees relative distance. 2021core RMSNorm A simplified LayerNorm that only rescales by root-mean-square. 2019core SwiGLU (Gated Linear Units) Adds a gating branch to the feed-forward layer to improve quality. 2020coreModern efficiency components
Grouped-Query Attention (GQA) Groups heads to share K/V, a middle ground between MHA and MQA. 2023core Multi-head Latent Attention (MLA) Caches a compressed low-rank latent instead of full keys and values. 2024core FlashAttention (v1–v3) Exact attention computed in GPU-memory-aware tiles — faster and lighter. 2022core KV Cache Stores past keys and values so each decoding step doesn't recompute them. core Temperature & Nucleus (Top-p) Sampling Controls randomness by sampling only from the top-p probability mass. 2019coreScaling up
Scaling Laws Loss falls as a smooth power law in parameters, data and compute. 2020core Chinchilla Compute-Optimal Scaling At fixed compute, scale parameters and tokens together (~20 tokens/param). 2022core Sparsely-Gated Mixture of Experts A gating network routes each input to only a few expert sub-networks. 2017core DeepSeekMoE (Fine-Grained & Shared Experts) Finer-grained experts plus always-on shared experts for specialization. 2024core Mixed Precision & FP8 Training Compute in 16- or 8-bit floats while keeping FP32 master weights. 2017core ZeRO / FSDP (Sharded Data Parallelism) Shards optimizer states, gradients and parameters across GPUs. 2019core
02Course units for this path
Goals, projects and self-tests that go with these algorithms.
U40 Tokenization & Embeddings BPE tokenisation, per-token pricing, embeddings and semantic similarity. 6 h6 algorithms2 paperslesson ready U41 Transformers & Attention Self-attention, multi-head attention, positional encoding, residuals and normalisation — build a GPT from scratch. 16 h6 algorithms3 paperslesson ready U42 Modern LLM Architecture RoPE, SwiGLU, RMSNorm, GQA/MLA, FlashAttention, sliding windows and context extension. 12 h12 algorithms5 papers U44 Pretraining, Data & Scaling Laws Pre-training data pipelines, scaling laws, Chinchilla, μP, MTP and training stability. 10 h6 algorithms4 papers U60 Decoding, Sampling & KV Cache Prefill and decode, temperature/top-p, KV cache and the memory-bandwidth bottleneck. 6 h4 algorithms2 paperslesson ready
03Other paths
- 1Zero to Deep Learning · From linear regression all the way to ResNet and the precursors of the Transformer — build intuition for how models are trained.24
- 2Post-training & Reasoning Models · From instruction tuning and LoRA to RLHF and DPO, then GRPO, RLVR and DeepSeek-R1 — how ChatGPT and o1/R1 are made.19
- 3AI Application Engineer · For product engineers: retrieval, RAG, agents, inference serving and security — turn AI into reliable applications fast.20
- 4Generative Models & Frontier Research · Diffusion/flow matching, multimodality, new architectures, world models and interpretability — the most active research areas of 2024–2026.26