// ▲ Algorithms Atlas
Pretraining, Scaling & Distributed Training
Scaling laws, hyperparameter transfer, mixed precision and parallelism — how frontier models get trained.
8 core1 extended2017–2024
TIMELINE
01Core algorithms
Learn these first. Sorted by year.
Mixed Precision & FP8 Training Compute in 16- or 8-bit floats while keeping FP32 master weights. 2017core Pipeline Parallelism (GPipe) Splits layers across devices and streams micro-batches like an assembly line. 2018core ZeRO / FSDP (Sharded Data Parallelism) Shards optimizer states, gradients and parameters across GPUs. 2019core Tensor Parallelism (Megatron-LM) Splits a layer's matrix multiplications across GPUs. 2019core Scaling Laws Loss falls as a smooth power law in parameters, data and compute. 2020core Chinchilla Compute-Optimal Scaling At fixed compute, scale parameters and tokens together (~20 tokens/param). 2022core Emergent Abilities Some abilities appear abruptly only past a certain scale. 2022core Multi-Token Prediction (MTP) Trains the model to predict several future tokens at once. 2024core
02Extended
Variants and follow-ups worth knowing once the core is clear.
03Course units that teach these
U44 Pretraining, Data & Scaling Laws Pre-training data pipelines, scaling laws, Chinchilla, μP, MTP and training stability. 10 h6 algorithms4 papers U64 GPUs, Distributed Training & AI Chips Memory, bandwidth and compute; mixed precision; data/tensor/pipeline parallelism; ZeRO/FSDP; the chip landscape. 10 h6 algorithms4 papers U90 Evaluation & Benchmarks Major benchmarks, arenas, data contamination and the limits of capability evaluation. 6 h2 algorithms2 paperslesson ready