// ◎ Transformers & Attention

Sliding-Window & Sparse Attention

Each token attends to a local window plus a few global tokens for linear cost.

滑动窗口/稀疏注意力 2020 Intermediate core

IN PLAIN WORDS

Full attention is quadratic. Longformer lets most tokens see a local window and a few global tokens see everything. Mistral 7B and Gemma interleave sliding-window and global layers.

Used in: Mistral 7B, Gemma 2/3

TIMELINE · prerequisites → this → what builds on it

2017 Transformer &…2020 Sliding-Window & …2023 Attention Sin…2025 Native Sparse…

01Original paper

Longformer: The Long-Document Transformer

Iz Beltagy, Matthew E. Peters, Arman Cohan · 2020

arXiv ↗ ✓ Verified · arXiv API

02Where it sits

Learn the left column first; the right column builds on this one.

Learn first

Transformer & Self-Attention

Sliding-Window & Sparse Attention

Then you can learn

Left: prerequisites · right: builds on this · dashed: related. Full map →

03Learning resources

Official docs, explainers, courses and reference code. All links checked.

Explainers

Lilian Weng：The Transformer Family v2 ↗EN

04Where the course teaches it

U42 Modern LLM Architecture RoPE, SwiGLU, RMSNorm, GQA/MLA, FlashAttention, sliding windows and context extension. 12 h12 algorithms5 papers

For AI / data

Verification: verified, checked 2026-10-21.
JSON: /api/learn/algorithms.json (id: sliding-window)