// ∿ New Architectures: SSM & Linear Attention

RWKV

Trains in parallel like a Transformer and infers like an RNN.

2023 Advanced core

IN PLAIN WORDS

Replaces softmax attention with a time-decayed linear mechanism: parallel training, fixed-size state at inference. Community-driven, with many versions.

Used in: RWKV open models

TIMELINE · prerequisites → this → what builds on it

1997 Long Short-Te…2020 Linear Attent…2023 RWKV

01Original paper

RWKV: Reinventing RNNs for the Transformer Era

Bo Peng, Eric Alcaide, Quentin Anthony, et al. · 2023

arXiv ↗ ✓ Verified · arXiv API

02Where it sits

Learn the left column first; the right column builds on this one.

Learn first

RWKV

Then you can learn

Nothing in the atlas builds on it yet

Left: prerequisites · right: builds on this · dashed: related. Full map →

03Learning resources

Official docs, explainers, courses and reference code. All links checked.

Reference code

github.com ↗EN

04Where the course teaches it

U43 MoE & Beyond-Transformer Architectures Sparse experts, load balancing, Mamba/SSMs, linear attention and hybrid architectures, diffusion language models. 10 h18 algorithms5 papers

For AI / data

Verification: verified, checked 2026-10-21.
JSON: /api/learn/algorithms.json (id: rwkv)