// ▲ Pretraining, Scaling & Distributed Training
ZeRO / FSDP (Sharded Data Parallelism)
Shards optimizer states, gradients and parameters across GPUs.
ZeRO / FSDP 数据并行 2019 Advanced core
IN PLAIN WORDS
Plain data parallelism replicates everything per GPU. ZeRO's three stages shard states, gradients and parameters, fetching them on demand. PyTorch FSDP is the equivalent.
Used in: DeepSpeed, PyTorch FSDP
01Original paper
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, et al. · 2019
arXiv ↗ ✓ Verified · arXiv API
02Where it sits
Learn the left column first; the right column builds on this one.
Learn first
Then you can learn
- Nothing in the atlas builds on it yet
Related:Tensor Parallelism (Megatron-LM)Pipeline Parallelism (GPipe)
Left: prerequisites · right: builds on this · dashed: related. Full map →
03Learning resources
Official docs, explainers, courses and reference code. All links checked.
04Where the course teaches it
On learning paths:Understanding an LLM End to End
For AI / data
- Verification: verified, checked 2026-10-21.
- JSON: /api/learn/algorithms.json (id:
zero)