// ▲ Pretraining, Scaling & Distributed Training
μP (Maximal Update Parametrization)
Tune hyperparameters on a small model and transfer them to a large one.
μP 超参迁移 2022 Research frontier extended
IN PLAIN WORDS
Scale initialization and learning rates with width by specific rules so optimal hyperparameters stay fixed across sizes — tune small, train big.
Used in: LLM hyperparameter search (e.g., Cerebras-GPT)
01Original paper
Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
Greg Yang, Edward J. Hu, Igor Babuschkin, et al. · 2022
arXiv ↗ ✓ Verified · arXiv API
02Where it sits
Learn the left column first; the right column builds on this one.
Learn first
Then you can learn
- Nothing in the atlas builds on it yet
Related:Chinchilla Compute-Optimal Scaling
Left: prerequisites · right: builds on this · dashed: related. Full map →
03Learning resources
Official docs, explainers, courses and reference code. All links checked.
Reference code
04Where the course teaches it
For AI / data
- Verification: verified, checked 2026-10-21.
- JSON: /api/learn/algorithms.json (id:
mup)