// ✳ Algorithms Atlas
Mixture of Experts
Sparse models that activate only some parameters, growing capacity without proportional compute.
5 core1 extended2017–2024
TIMELINE
01Core algorithms
Learn these first. Sorted by year.
Sparsely-Gated Mixture of Experts A gating network routes each input to only a few expert sub-networks. 2017core Switch Transformer A simplified MoE that routes each token to a single expert. 2021core Mixtral (Top-2 MoE) An open MoE that picks 2 of 8 experts per token. 2024core DeepSeekMoE (Fine-Grained & Shared Experts) Finer-grained experts plus always-on shared experts for specialization. 2024core Auxiliary-Loss-Free Load Balancing Balances expert load with dynamic biases instead of an extra loss. 2024core
02Extended
Variants and follow-ups worth knowing once the core is clear.
03Course units that teach these
U43 MoE & Beyond-Transformer Architectures Sparse experts, load balancing, Mamba/SSMs, linear attention and hybrid architectures, diffusion language models. 10 h18 algorithms5 papers U64 GPUs, Distributed Training & AI Chips Memory, bandwidth and compute; mixed precision; data/tensor/pipeline parallelism; ZeRO/FSDP; the chip landscape. 10 h6 algorithms4 papers