// ◎ Transformers & Attention
Linear Attention
Approximates softmax with kernels to make attention linear in length.
线性注意力 aka Gated Linear Attention / GLA 2020 Advanced core
IN PLAIN WORDS
Rewrite softmax(QKᵀ)V as φ(Q)(φ(K)ᵀV); computing the right side first is linear and admits an RNN-style recurrence. The starting point for RetNet and Gated DeltaNet.
Used in: New architectures and hybrids; i14’s Laplace 3B uses gated linear attention in every layer with a fixed 29.5 MB recurrent state per sequence (AWS blog, 20 Oct 2026); KronLSTM (arXiv 2610.26834, Oct 2026) adds a Kronecker delta update
TIMELINE · prerequisites → this → what builds on it
01Original paper
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, et al. · 2020
arXiv ↗ ✓ Verified · arXiv API
02Where it sits
Learn the left column first; the right column builds on this one.
Learn first
Related:Mamba (Selective State Spaces)
Left: prerequisites · right: builds on this · dashed: related. Full map →
03Learning resources
Official docs, explainers, courses and reference code. All links checked.
Reference code
Explainers
04Where the course teaches it
On learning paths:Generative Models & Frontier Research
For AI / data
- Verification: verified, checked 2026-10-22.
- JSON: /api/learn/algorithms.json (id:
linear-attention)