// ◎ Transformers & Attention
Transformer & Self-Attention
An attention-only, parallelizable sequence model — the backbone of all LLMs.
Transformer 与自注意力 2017 Foundational core
IN PLAIN WORDS
Each token emits a query, key and value; softmax(QKᵀ)·V lets every token attend to all others. Multi-head attention, FFN, residuals and normalization stack into layers trained fully in parallel.
Used in: Every LLM: GPT, Claude, Gemini, Llama, Qwen, DeepSeek
TIMELINE · prerequisites → this → what builds on it
01Original paper
Attention Is All You Need
Ashish Vaswani, Noam Shazeer, Niki Parmar, et al. · 2017
arXiv ↗ ✓ Verified · arXiv API
02Where it sits
Learn the left column first; the right column builds on this one.
Then you can learn · 29
- SwiGLU (Gated Linear Units)
- BERT
- T5 (Text-to-Text Transfer Transformer)
- GPT-3 & In-Context Learning
- Rotary Position Embedding (RoPE)
- ALiBi (Attention with Linear Biases)
- Multi-Query Attention (MQA)
- FlashAttention (v1–v3)
Show 21 more
- Sliding-Window & Sparse Attention
- Linear Attention
- Sparsely-Gated Mixture of Experts
- Multi-Token Prediction (MTP)
- Tensor Parallelism (Megatron-LM)
- KV Cache
- Temperature & Nucleus (Top-p) Sampling
- Visual Autoregressive Modeling (VAR)
- Vision Transformer (ViT)
- Flamingo
- Whisper
- Genie (Generative Interactive Environments)
- AlphaFold 2
- AlphaGeometry (Neuro-Symbolic Reasoning)
- Logit Lens & Probing
- Circuits & Induction Heads
- Locating & Editing Facts (ROME)
- Looped Language Models
- DETR (Detection Transformer)
- Decision Transformer
- wav2vec 2.0
Left: prerequisites · right: builds on this · dashed: related. Full map →
03Learning resources
Official docs, explainers, courses and reference code. All links checked.
Explainers
Video
Reference code
Courses
04Where the course teaches it
On learning paths:Understanding an LLM End to End
For AI / data
- Verification: verified, checked 2026-10-21.
- JSON: /api/learn/algorithms.json (id:
transformer)