Skip to content
English

Switch theme

Discover▾

Not sure where to start? This week’s hottest software, models and stacks in three minutes.

AI Apps▾

AI software people use: from assistants like ChatGPT to tools for coding, video and workflows, picked by purpose.

For Agents▾

Software for AI: tools that let your agent (Claude Code, Codex, Cursor…) browse, read docs, run code and remember.

Models▾

Choose models, estimate cost, self-host: prices, rankings, benchmarks, free tiers and compute.

Solutions▾

A goal or an error message? Ready-made stacks, steps and fixes by scenario.

Learn▾

A systematic course from zero to the frontier, with an algorithm library, must-read papers and tutorials.

Frontier▾

Daily AI news, weekly highlights, the latest papers and how AI is changing every discipline.

// ✳ Mixture of Experts

GShard (Expert Parallelism)

Shards MoE experts across thousands of accelerators.

GShard 专家并行 2020 Advanced extended

Original paper ↓ What to learn first →

IN PLAIN WORDS

Each device hosts some experts; tokens are sent to them via all-to-all and returned. It established expert parallelism and capacity factors.

Used in: Large-scale MoE training frameworks

TIMELINE · prerequisites → this → what builds on it

2017 Sparsely-Gate…2020 GShard (Expert Pa…

01Original paper

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, et al. · 2020

arXiv ↗ ✓ Verified · arXiv API

02Where it sits

Learn the left column first; the right column builds on this one.

Learn first

GShard (Expert Parallelism)

Then you can learn

  • Nothing in the atlas builds on it yet

Related:Switch TransformerZeRO / FSDP (Sharded Data Parallelism)

Sparsely-Gated …Switch Transfor…ZeRO / FSDP (Sh… GShard (Expert Para…

Left: prerequisites · right: builds on this · dashed: related. Full map →

03Learning resources

Official docs, explainers, courses and reference code. All links checked.

Explainers

Reference code

04Where the course teaches it

U43 MoE & Beyond-Transformer Architectures Sparse experts, load balancing, Mamba/SSMs, linear attention and hybrid architectures, diffusion language models. 10 h18 algorithms5 papers U64 GPUs, Distributed Training & AI Chips Memory, bandwidth and compute; mixed precision; data/tensor/pipeline parallelism; ZeRO/FSDP; the chip landscape. 10 h6 algorithms4 papers
For AI / data

Data updated: AI algorithms & curriculum(UTC+8)