// ⬡ Deep Learning Foundations
RMSNorm
A simplified LayerNorm that only rescales by root-mean-square.
2019 Foundational core
IN PLAIN WORDS
Drops mean-centering and bias, only dividing by RMS — cheaper with similar quality. Used by nearly all modern open LLMs.
Used in: Llama, Qwen, DeepSeek and others
01Original paper
Root Mean Square Layer Normalization
Biao Zhang, Rico Sennrich · 2019
arXiv ↗ ✓ Verified · arXiv API
02Where it sits
Learn the left column first; the right column builds on this one.
Related:SwiGLU (Gated Linear Units)
Left: prerequisites · right: builds on this · dashed: related. Full map →
03Learning resources
Official docs, explainers, courses and reference code. All links checked.
04Where the course teaches it
U31 Training Deep Nets SGD/momentum/Adam/AdamW/Muon, learning-rate schedules, BatchNorm/LayerNorm/RMSNorm, dropout and residuals. 10 h10 algorithms4 papers U42 Modern LLM Architecture RoPE, SwiGLU, RMSNorm, GQA/MLA, FlashAttention, sliding windows and context extension. 12 h12 algorithms5 papers
On learning paths:Understanding an LLM End to End
For AI / data
- Verification: verified, checked 2026-10-22.
- JSON: /api/learn/algorithms.json (id:
rmsnorm)