// ◎ Transformers & Attention
FlashAttention (v1–v3)
Exact attention computed in GPU-memory-aware tiles — faster and lighter.
FlashAttention 2022 Advanced core
IN PLAIN WORDS
Attention is memory-bound, not compute-bound. FlashAttention tiles Q/K/V into on-chip SRAM with an online softmax and never materializes the full matrix. v2 improves parallelism; v3 uses Hopper async and FP8.
Used in: PyTorch SDPA, vLLM, most training/inference stacks
TIMELINE · prerequisites → this → what builds on it
01Original paper
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Tri Dao, Daniel Y. Fu, Stefano Ermon, et al. · 2022
arXiv ↗ ✓ Verified · arXiv API
Note: v2：arXiv:2307.08691（2023）；v3：arXiv:2407.08608（2024）
02Where it sits
Learn the left column first; the right column builds on this one.
Learn first
Then you can learn
Related:PagedAttention (vLLM)
Left: prerequisites · right: builds on this · dashed: related. Full map →
03Learning resources
Official docs, explainers, courses and reference code. All links checked.
Reference code
Official docs
Books
04Where the course teaches it
On learning paths:Understanding an LLM End to End
For AI / data
- Verification: verified, checked 2026-10-21.
- JSON: /api/learn/algorithms.json (id:
flashattention)