// ▲ Pretraining, Scaling & Distributed Training
Pipeline Parallelism (GPipe)
Splits layers across devices and streams micro-batches like an assembly line.
流水线并行 2018 Advanced core
IN PLAIN WORDS
Each device owns some layers and micro-batches flow through to reduce idle bubbles. Combined with tensor and data parallelism it forms 3D parallelism.
Used in: Training of very large models
01Original paper
GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
Yanping Huang, Youlong Cheng, Ankur Bapna, et al. · 2018
arXiv ↗ ✓ Verified · arXiv API
02Where it sits
Learn the left column first; the right column builds on this one.
Learn first
Then you can learn
- Nothing in the atlas builds on it yet
Related:ZeRO / FSDP (Sharded Data Parallelism)
Left: prerequisites · right: builds on this · dashed: related. Full map →
03Learning resources
Official docs, explainers, courses and reference code. All links checked.
04Where the course teaches it
For AI / data
- Verification: verified, checked 2026-10-21.
- JSON: /api/learn/algorithms.json (id:
pipeline-parallel)