Compute & Hardware
Distributed Training and Clusters
Data, tensor and pipeline parallelism, interconnects and failures at scale.
Expert 20 min For: Researchers ✎ In progress
✎ This lesson is still being written. Below is the planned outline; it is not indexed by search engines yet.
What this lesson will cover
- Data parallelism and ZeRO
- Tensor and pipeline parallelism
- Interconnects
- Checkpointing and failures