// ▲ Pretraining, Scaling & Distributed Training
Tensor Parallelism (Megatron-LM)
Splits a layer's matrix multiplications across GPUs.
张量并行（Megatron） 2019 Advanced core
IN PLAIN WORDS
Split attention heads and FFN weights by columns/rows so each GPU computes a part, then combine with a few collectives. Best within a fast-interconnect node.
Used in: Megatron-LM, NeMo
TIMELINE · prerequisites → this → what builds on it
01Original paper
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, et al. · 2019
arXiv ↗ ✓ Verified · arXiv API
02Where it sits
Learn the left column first; the right column builds on this one.
Learn first
Then you can learn
Related:ZeRO / FSDP (Sharded Data Parallelism)
Left: prerequisites · right: builds on this · dashed: related. Full map →
03Learning resources
Official docs, explainers, courses and reference code. All links checked.
04Where the course teaches it
For AI / data
- Verification: verified, checked 2026-10-21.
- JSON: /api/learn/algorithms.json (id:
tensor-parallel)