// Stage 4 · Transformers & LLMs

MoE & Beyond-Transformer Architectures

Sparse experts, load balancing, Mamba/SSMs, linear attention and hybrid architectures, diffusion language models.

What you will learn ↓ Code & exercises ↓

about 10 hours18 algorithms5 paperslesson text in progress

01Goals

Understand top-k routing and expert parallelism
Distinguish total and active parameters
Compare the complexity of Transformers and SSMs

Do these first: Modern LLM Architecture

02Algorithms to master

Each card has the original paper, prerequisites and resources.

Sparsely-Gated Mixture of Experts A gating network routes each input to only a few expert sub-networks. 2017core GShard (Expert Parallelism) Shards MoE experts across thousands of accelerators. 2020ext Switch Transformer A simplified MoE that routes each token to a single expert. 2021core Mixtral (Top-2 MoE) An open MoE that picks 2 of 8 experts per token. 2024core DeepSeekMoE (Fine-Grained & Shared Experts) Finer-grained experts plus always-on shared experts for specialization. 2024core Auxiliary-Loss-Free Load Balancing Balances expert load with dynamic biases instead of an extra loss. 2024core Structured State Spaces (S4) Models very long sequences efficiently with structured state-space equations. 2021ext Mamba (Selective State Spaces) Input-dependent state spaces with linear-time cost and Transformer-like quality. 2023core Mamba-2 (State Space Duality) Shows SSMs and a class of attention are dual, yielding faster Mamba. 2024ext Linear Attention Approximates softmax with kernels to make attention linear in length. 2020core RWKV Trains in parallel like a Transformer and infers like an RNN. 2023core Retentive Network (RetNet) An attention alternative with parallel, recurrent and chunkwise modes. 2023ext Hyena Hierarchy Replaces attention with implicitly parameterized long convolutions and gating. 2023ext xLSTM Upgrades LSTM with exponential gating and matrix memory. 2024ext Gated Delta Networks Linear attention combining delta-rule memory updates with gated forgetting. 2024core Hybrid Architectures (Jamba) Interleaves attention and Mamba layers (plus MoE) in one model. 2024core Diffusion Language Models (LLaDA) Generates text by iteratively unmasking a fully masked sequence. 2025core Looped Language Models Re-applies the same layers several times, trading depth in parameters for depth in compute. 2018ext

03Must-read papers

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer ↗2017 ✓ verified in Essentials
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity ↗2021 ✓ verified
DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models ↗2024 ✓ verified
Mamba: Linear-Time Sequence Modeling with Selective State Spaces ↗2023 ✓ verified in Essentials
Gated Delta Networks: Improving Mamba2 with Delta Rule ↗2024 ✓ verified

04Courses & videos

Mixture of Experts Explained ↗ExplainersEN
Get on the SSM train ↗ExplainersEN
The Annotated S4 ↗ExplainersEN
fla-org/flash-linear-attention ↗Reference codeEN

05Hands-on: project, code & exercises

Free, official or author-published pages. Most run in the browser or on a free Colab GPU.

PROJECT

Implement an 8-expert, top-2 MoE feed-forward layer to replace your small GPT’s FFN and watch the expert load distribution.

1
fla-org/flash-linear-attention ↗Reference code
Swap a small model’s attention for linear-attention / Gated DeltaNet layers and compare.
2
github.com ↗Reference code
Install Mamba, run the official example and compare inference speed with a Transformer.

06Self-test

Answer in your head, then open to check.

Why does MoE save compute but cost memory at inference?

Only a few experts run, but all expert weights must be loaded.

Why does Mamba need no KV cache at inference?

It uses a fixed-size recurrent state.

Why do hybrid architectures keep a few attention layers?

They provide precise retrieval and copying.

On these routes: Algorithm Researchers