{ "id": "transformer-attention", "module": "llm-core", "level": 2, "minutes": 25, "audiences": [ "beginner", "developer", "researcher" ], "software": [ "llama-cpp", "mlx-lm" ], "text": { "en": { "title": "Transformers and Attention Explained: Q, K, V, Multi-Head and the Decoder Block", "description": "How self-attention works step by step (queries, keys, values), why it uses softmax and scaling, what multi-head attention and causal masking do, and how a full Transformer layer is built.", "updated": "2026-10-21", "takeaways": [ "Self-attention lets every token build its new representation as a weighted mix of the other tokens, with weights computed from query–key similarity.", "Attention(Q, K, V) = softmax(QKᵀ / √dₖ) · V; multi-head attention runs several of these in parallel to capture different relationships.", "GPT-style LLMs are decoder-only Transformers: a causal mask stops each token from seeing the future, which is what makes next-token training possible.", "A Transformer layer = attention + feed-forward network, each wrapped in a residual connection and normalisation, stacked dozens of times." ], "faq": [ { "q": "Why is it called “attention”?", "a": "Because for each token the model computes how much to “attend to” every other token — a set of weights that sum to 1 — and then reads information mostly from the tokens with high weights." }, { "q": "Why did Transformers replace RNNs?", "a": "RNNs process tokens one after another, which is slow to train and struggles with long-range dependencies. Attention connects all positions directly and can be computed in parallel on GPUs, so Transformers scale to far more data and parameters." }, { "q": "Why does attention get expensive for long contexts?", "a": "Every token is compared with every other token, so compute and memory for the attention scores grow with the square of the sequence length. Techniques such as FlashAttention, grouped-query attention and KV caching reduce the practical cost." } ], "sources": [ { "title": "Vaswani et al. (2017), Attention Is All You Need", "url": "https://arxiv.org/abs/1706.03762" }, { "title": "Bahdanau, Cho & Bengio (2014), Neural Machine Translation by Jointly Learning to Align and Translate", "url": "https://arxiv.org/abs/1409.0473" }, { "title": "Jay Alammar, The Illustrated Transformer", "url": "https://jalammar.github.io/illustrated-transformer/" }, { "title": "Harvard NLP, The Annotated Transformer", "url": "https://nlp.seas.harvard.edu/annotated-transformer/" }, { "title": "Radford et al. (2019), Language Models are Unsupervised Multitask Learners (GPT-2)", "url": "https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf" }, { "title": "Su et al. (2021), RoFormer: Enhanced Transformer with Rotary Position Embedding", "url": "https://arxiv.org/abs/2104.09864" }, { "title": "Dao et al. (2022), FlashAttention", "url": "https://arxiv.org/abs/2205.14135" } ], "url": "https://aimap.page/en/learn/llm-core/transformer-attention/", "markdown": "## The problem attention solves\n\nTake the sentence *\"The animal didn't cross the street because **it** was too tired.\"* To understand **it**, a model must connect it to *animal*, not *street*. Change *tired* to *wide* and the answer flips. Meaning depends on other words, sometimes far away.\n\nEarlier models (RNNs, LSTMs) read text one token at a time and squeezed everything into a running memory, which made long-range links hard and training slow. Attention was first added on top of such models for translation ([Bahdanau et al., 2014](https://arxiv.org/abs/1409.0473)); the **Transformer** ([Vaswani et al., 2017](https://arxiv.org/abs/1706.03762)) went further and built the whole model out of attention — hence the title *Attention Is All You Need*.\n\n## Self-attention in four steps\n\nEach token starts as a vector (its embedding, see [Tokens and Embeddings](/learn/tokens-embeddings)). For every token, the layer computes three new vectors by multiplying with learned matrices:\n\n- **Query (Q)** – \"what am I looking for?\"\n- **Key (K)** – \"what do I contain?\"\n- **Value (V)** – \"what information do I pass on if selected?\"\n\nThen:\n\n1. **Score**: compare the token's query with every token's key using a dot product. A high score means \"relevant\".\n2. **Scale**: divide by √dₖ (the square root of the key size) to keep scores in a range where training is stable.\n3. **Normalise**: apply **softmax** so the scores become positive weights that sum to 1.\n4. **Mix**: the token's output is the weighted sum of all value vectors.\n\nIn matrix form, for all tokens at once:\n\n`Attention(Q, K, V) = softmax(Q·Kᵀ / √dₖ) · V`\n\n```mermaid\nflowchart LR\n X[\"Token vectors\"] --> Q[\"Q = X·Wq\"]\n X --> K[\"K = X·Wk\"]\n X --> V[\"V = X·Wv\"]\n Q --> S[\"Scores = Q·Kᵀ / √dk\"]\n K --> S\n S --> M[\"Causal mask (hide future tokens)\"]\n M --> SM[\"softmax → attention weights\"]\n SM --> O[\"Weights · V = new token vectors\"]\n V --> O\n```\n\nFor *it* in our sentence, a well-trained head gives *animal* a large weight, so the new vector for *it* carries information about the animal.\n\n## Multi-head attention\n\nOne set of weights can only express one kind of relationship at a time. **Multi-head attention** runs several attention operations (\"heads\") in parallel, each with its own Q/K/V matrices, then concatenates their outputs. The original Transformer used 8 heads; large LLMs use dozens. Different heads end up tracking different things — previous token, syntactic subject, matching brackets, repeated phrases ([What LLMs Learn Inside](/learn/inside-llms)).\n\n## Where does word order come in?\n\nAttention by itself treats the input as a set: shuffling tokens would not change the scores. Transformers therefore add **positional information**. The 2017 paper added fixed sine/cosine patterns to the embeddings; most current LLMs use **rotary position embeddings (RoPE)** ([Su et al., 2021](https://arxiv.org/abs/2104.09864)), which rotate Q and K vectors according to position so that scores depend on relative distance.\n\n## The decoder-only Transformer (GPT-style)\n\nThe original Transformer had an *encoder* (reads the source sentence) and a *decoder* (writes the translation). GPT models ([Radford et al., 2019](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)) and almost all chat LLMs keep only the **decoder** stack, with a **causal mask**: a token may attend only to itself and earlier tokens. That lets one pass over a training text produce a next-token prediction at *every* position at once — extremely efficient training.\n\nOne **Transformer layer (block)** is:\n\n1. Normalisation → **masked multi-head self-attention** → add the input back (residual connection).\n2. Normalisation → **feed-forward network** (two linear layers with a non-linearity, applied to each token separately) → residual connection.\n\n```mermaid\nflowchart TB\n IN[\"Token vectors (embeddings + position)\"] --> N1[\"Norm\"] --> A[\"Masked multi-head attention\"] --> R1((\"+\"))\n IN --> R1\n R1 --> N2[\"Norm\"] --> F[\"Feed-forward network\"] --> R2((\"+\"))\n R1 --> R2\n R2 --> NEXT[\"… repeat for N layers …\"] --> HEAD[\"Final norm → linear → softmax over vocabulary\"]\n```\n\nStack this block dozens of times (GPT-3's largest version had 96 layers), add a final layer that turns the last vector into probabilities over the vocabulary, and you have an LLM. The residual connections act as a shared \"stream\" that each layer reads from and writes into.\n\n## The cost of attention\n\nBecause every token is compared with every other token, the score matrix grows with the **square** of the sequence length. Doubling the context roughly quadruples that part of the work. Engineering advances — **FlashAttention** ([Dao et al., 2022](https://arxiv.org/abs/2205.14135)), grouped-query attention, the **KV cache** for generation ([How LLMs Generate Text](/learn/llm-inference)) — are why million-token context windows became practical.\n\n## See it run\n\nSmall open models run on a laptop with [llama.cpp](/apps/llama-cpp) or Apple's [MLX LM](/apps/mlx-lm); their model cards list the number of layers, heads and the hidden size — the exact quantities described here. For visual walkthroughs, Jay Alammar's [Illustrated Transformer](https://jalammar.github.io/illustrated-transformer/) and Harvard's [Annotated Transformer](https://nlp.seas.harvard.edu/annotated-transformer/) (with code) are excellent next steps.\n\n## Algorithm cards\n\n| Algorithm | In one line | Original paper |\n|---|---|---|\n| Transformer & self-attention | An attention-only, parallelisable sequence model — the backbone of all LLMs | [Vaswani et al., 2017](https://arxiv.org/abs/1706.03762) |\n| BERT | A bidirectional Transformer encoder pre-trained by masked-word prediction | [Devlin et al., 2018](https://arxiv.org/abs/1810.04805) |\n| T5 | Casts every NLP task as text in, text out | [Raffel et al., 2019](https://arxiv.org/abs/1910.10683) |\n| GPT-3 & in-context learning | Large enough models learn new tasks from a few examples in the prompt | [Brown et al., 2020](https://arxiv.org/abs/2005.14165) |\n| Rotary position embedding (RoPE) | Rotates query/key vectors so attention sees relative distance | [Su et al., 2021](https://arxiv.org/abs/2104.09864) |\n| Layer normalisation | Normalises within each example instead of across the batch | [Ba et al., 2016](https://arxiv.org/abs/1607.06450) |\n\n## Must-read papers\n\n1. **Attention Is All You Need** (2017) — introduces the Transformer; everything in this lesson comes from it. [arXiv:1706.03762](https://arxiv.org/abs/1706.03762)\n2. **BERT** (2018) — the encoder-only, bidirectional route; the best contrast for \"encoder vs decoder\". [arXiv:1810.04805](https://arxiv.org/abs/1810.04805)\n3. **Language Models are Few-Shot Learners** (GPT-3, 2020) — scaling a decoder-only Transformer yields in-context learning. [arXiv:2005.14165](https://arxiv.org/abs/2005.14165)\n\n## Hands-on project (about 16 hours)\n\nFollow [nanoGPT](https://github.com/karpathy/nanoGPT) to train a character-level GPT on Shakespeare (or any small corpus), then plot one layer's attention weights as a heat map to see what different heads attend to. For the code walk-through use [The Annotated Transformer](https://nlp.seas.harvard.edu/annotated-transformer/) and Karpathy's [Let's reproduce GPT-2 (124M)](https://www.youtube.com/watch?v=l8pRSuU81PU); for a full course, [Stanford CS224N](https://web.stanford.edu/class/cs224n/).\n\n## Self-check\n\n
1. Why divide attention scores by √dₖ?\n\nDot products grow with dimension, pushing softmax to extremes (almost one-hot) and killing gradients; dividing by √dₖ keeps scores in a workable range.\n
\n\n
2. What does the causal mask do?\n\nEach position can only see earlier tokens, never the \"answer\", which is what makes next-token training possible and matches one-token-at-a-time generation.\n
\n\n
3. Why multi-head rather than single-head attention?\n\nEach head computes attention in its own subspace, so the model can track several kinds of relation at once (e.g. syntax and coreference) and combine them.\n
\n\n
4. If the context doubles, roughly how much more work is the attention-score matrix?\n\nAbout 4×, because it grows with the square of sequence length.\n
" }, "zh-hans": { "title": "Transformer 与注意力机制详解：Q、K、V、多头注意力与解码器结构", "description": "一步步讲清自注意力（查询、键、值）的计算，为什么要用 softmax 和缩放，多头注意力和因果掩码的作用，以及一个完整的 Transformer 层是怎么搭起来的。", "updated": "2026-10-21", "takeaways": [ "自注意力让每个 token 把其他 token 的信息按权重混合，生成自己的新表示；权重由查询和键的相似度算出。", "Attention(Q, K, V) = softmax(QKᵀ / √dₖ) · V；多头注意力并行运行多组这样的计算，捕捉不同类型的关系。", "GPT 类大模型是纯解码器 Transformer：因果掩码让每个 token 看不到后面的内容，这正是“预测下一个 token”训练的前提。", "一个 Transformer 层 = 注意力 + 前馈网络，各自包着残差连接和归一化，再叠几十层。" ], "faq": [ { "q": "为什么叫“注意力”？", "a": "因为模型会为每个 token 计算它应该“关注”其他每个 token 多少，得到一组加起来等于 1 的权重，再主要从权重高的 token 那里读取信息。" }, { "q": "Transformer 为什么取代了 RNN？", "a": "RNN 必须一个接一个地处理 token，训练慢，也难以捕捉远距离依赖。注意力把所有位置直接连起来，还能在 GPU 上并行计算，所以 Transformer 能扩展到大得多的数据和参数规模。" }, { "q": "为什么上下文越长，注意力越贵？", "a": "每个 token 都要和其他所有 token 比较，所以注意力分数的计算量和内存随序列长度的平方增长。FlashAttention、分组查询注意力和 KV Cache 等技术降低了实际开销。" } ], "sources": [ { "title": "Vaswani 等（2017），Attention Is All You Need", "url": "https://arxiv.org/abs/1706.03762" }, { "title": "Bahdanau、Cho、Bengio（2014），Neural Machine Translation by Jointly Learning to Align and Translate", "url": "https://arxiv.org/abs/1409.0473" }, { "title": "Jay Alammar，The Illustrated Transformer（图解 Transformer）", "url": "https://jalammar.github.io/illustrated-transformer/" }, { "title": "Harvard NLP，The Annotated Transformer（带代码注解）", "url": "https://nlp.seas.harvard.edu/annotated-transformer/" }, { "title": "Radford 等（2019），Language Models are Unsupervised Multitask Learners（GPT-2）", "url": "https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf" }, { "title": "苏剑林等（2021），RoFormer: Enhanced Transformer with Rotary Position Embedding（RoPE）", "url": "https://arxiv.org/abs/2104.09864" }, { "title": "Dao 等（2022），FlashAttention", "url": "https://arxiv.org/abs/2205.14135" } ], "url": "https://aimap.page/zh-hans/learn/llm-core/transformer-attention/", "markdown": "## 注意力要解决什么问题\n\n看这句话：*“那只动物没有过马路，因为**它**太累了。”* 要理解“它”，模型得把它和“动物”而不是“马路”联系起来。把“太累了”换成“太宽了”，答案就反过来。词的含义取决于其他词，而且有时离得很远。\n\n早期模型（RNN、LSTM）一次读一个 token，把一切压进一个不断更新的记忆里，远距离的联系难以建立，训练也慢。注意力最早是加在这类模型上做翻译的（[Bahdanau 等，2014](https://arxiv.org/abs/1409.0473)）；**Transformer**（[Vaswani 等，2017](https://arxiv.org/abs/1706.03762)）更进一步，整个模型都用注意力搭建，论文标题就叫 *Attention Is All You Need*。\n\n## 自注意力四步走\n\n每个 token 一开始是一个向量（它的 embedding，见 [Token 与 Embedding](/learn/tokens-embeddings)）。对每个 token，这一层用三个学出来的矩阵分别算出三个新向量：\n\n- **查询（Query，Q）**：“我在找什么？”\n- **键（Key，K）**：“我身上有什么？”\n- **值（Value，V）**：“如果被选中，我传递什么信息？”\n\n然后：\n\n1. **打分**：用点积比较这个 token 的查询和每个 token 的键，分数高表示“相关”。\n2. **缩放**：除以 √dₖ（键向量长度的平方根），让分数保持在训练稳定的范围。\n3. **归一化**：用 **softmax** 把分数变成加起来等于 1 的正权重。\n4. **混合**：这个 token 的输出 = 所有值向量的加权和。\n\n写成矩阵形式，一次算出所有 token：\n\n`Attention(Q, K, V) = softmax(Q·Kᵀ / √dₖ) · V`\n\n```mermaid\nflowchart LR\n X[\"token 向量\"] --> Q[\"Q = X·Wq\"]\n X --> K[\"K = X·Wk\"]\n X --> V[\"V = X·Wv\"]\n Q --> S[\"分数 = Q·Kᵀ / √dk\"]\n K --> S\n S --> M[\"因果掩码（遮住后面的 token）\"]\n M --> SM[\"softmax → 注意力权重\"]\n SM --> O[\"权重 · V = 新的 token 向量\"]\n V --> O\n```\n\n对例句中的“它”，训练好的注意力头会给“动物”很大的权重，于是“它”的新向量就带上了关于那只动物的信息。\n\n## 多头注意力\n\n一组权重一次只能表达一种关系。**多头注意力**并行运行多个注意力计算（称为“头”），每个头有自己的 Q/K/V 矩阵，最后把输出拼起来。原始 Transformer 用 8 个头，大模型用几十个。不同的头会各自追踪不同的东西：前一个 token、句法主语、配对的括号、重复出现的短语等（见 [大模型内部学到了什么](/learn/inside-llms)）。\n\n## 词序从哪里来？\n\n注意力本身把输入当成一个集合：打乱 token 顺序，分数不会变。所以 Transformer 要额外加入**位置信息**。2017 年的论文把固定的正弦/余弦模式加到 embedding 上；如今多数大模型使用**旋转位置编码（RoPE）**（[苏剑林等，2021](https://arxiv.org/abs/2104.09864)），按位置旋转 Q 和 K 向量，让分数取决于相对距离。\n\n## 纯解码器 Transformer（GPT 式）\n\n原始 Transformer 有*编码器*（读原文）和*解码器*（写译文）。GPT 系列（[Radford 等，2019](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)）和几乎所有聊天大模型只保留了**解码器**，并加上**因果掩码**：每个 token 只能关注自己和前面的 token。这样，过一遍训练文本，就能同时在*每一个*位置上得到“下一个 token”的预测，训练效率极高。\n\n一个 **Transformer 层（block）** 包括：\n\n1. 归一化 → **带掩码的多头自注意力** → 把输入加回来（残差连接）。\n2. 归一化 → **前馈网络**（两层线性变换夹一个非线性，对每个 token 单独计算）→ 残差连接。\n\n```mermaid\nflowchart TB\n IN[\"token 向量（embedding + 位置）\"] --> N1[\"归一化\"] --> A[\"带掩码的多头注意力\"] --> R1((\"+\"))\n IN --> R1\n R1 --> N2[\"归一化\"] --> F[\"前馈网络\"] --> R2((\"+\"))\n R1 --> R2\n R2 --> NEXT[\"…… 重复 N 层 ……\"] --> HEAD[\"最终归一化 → 线性层 → 词表上的 softmax\"]\n```\n\n把这个层叠几十次（GPT-3 最大的版本有 96 层），最后加一层把末尾的向量变成词表上的概率，就是一个大模型。残差连接就像一条共享的“信息流”，每一层都从中读取、再往里写入。\n\n## 注意力的代价\n\n因为每个 token 都要和其他所有 token 比较，分数矩阵随序列长度的**平方**增长：上下文翻倍，这部分工作量大约变成四倍。**FlashAttention**（[Dao 等，2022](https://arxiv.org/abs/2205.14135)）、分组查询注意力、生成时使用的 **KV Cache**（见 [大模型如何生成文本](/learn/llm-inference)）等工程进展，才让百万 token 级别的上下文窗口变得可用。\n\n## 亲眼看看\n\n小型开源模型可以用 [llama.cpp](/apps/llama-cpp) 或苹果的 [MLX LM](/apps/mlx-lm) 在笔记本上运行；它们的模型卡会写明层数、注意力头数和隐藏层大小，正是本课讲的这些量。想看图解，推荐 Jay Alammar 的 [The Illustrated Transformer](https://jalammar.github.io/illustrated-transformer/) 和哈佛的 [The Annotated Transformer](https://nlp.seas.harvard.edu/annotated-transformer/)（附代码）。\n\n## 算法卡片\n\n| 算法 | 一句话 | 原始论文 |\n|---|---|---|\n| Transformer 与自注意力 | 完全基于注意力、可并行训练的序列模型，是所有大模型的骨架 | [Vaswani 等，2017](https://arxiv.org/abs/1706.03762) |\n| BERT | 双向 Transformer 编码器，通过“完形填空”预训练 | [Devlin 等，2018](https://arxiv.org/abs/1810.04805) |\n| T5 | 把所有 NLP 任务统一成“输入文本→输出文本” | [Raffel 等，2019](https://arxiv.org/abs/1910.10683) |\n| GPT-3 与上下文学习 | 足够大的模型只看几个例子就能完成新任务 | [Brown 等，2020](https://arxiv.org/abs/2005.14165) |\n| 旋转位置编码 RoPE | 旋转 Q/K 向量来编码位置，让注意力天然感知相对距离 | [Su 等，2021](https://arxiv.org/abs/2104.09864) |\n| 层归一化 LayerNorm | 在单个样本内部做标准化，不依赖 batch | [Ba 等，2016](https://arxiv.org/abs/1607.06450) |\n\n## 必读论文\n\n1. **Attention Is All You Need**（2017）——提出 Transformer，本课的全部内容都出自这篇。[arXiv:1706.03762](https://arxiv.org/abs/1706.03762)\n2. **BERT**（2018）——只用编码器、双向注意力的路线，理解“编码器 vs 解码器”的最好对照。[arXiv:1810.04805](https://arxiv.org/abs/1810.04805)\n3. **Language Models are Few-Shot Learners**（GPT-3，2020）——纯解码器 Transformer 放大后出现上下文学习。[arXiv:2005.14165](https://arxiv.org/abs/2005.14165)\n\n## 动手项目（约 16 学时）\n\n照着 [nanoGPT](https://github.com/karpathy/nanoGPT) 在莎士比亚或中文古诗语料上训练一个字符级 GPT，然后取出某一层的注意力权重画成热力图，看看不同注意力头分别在“看”什么。想先理解代码，可配合 [The Annotated Transformer](https://nlp.seas.harvard.edu/annotated-transformer/) 和 Karpathy 的 [Let's reproduce GPT-2 (124M)](https://www.youtube.com/watch?v=l8pRSuU81PU)；系统学习可看 [Stanford CS224N](https://web.stanford.edu/class/cs224n/)。\n\n## 自测\n\n
1. 注意力分数为什么要除以 √dₖ？\n\n维度越高，点积的数值越大，softmax 会变得极端（几乎只剩一个 1），梯度也随之消失；除以 √dₖ 把分数拉回合适的范围。\n
\n\n
2. 因果掩码有什么作用？\n\n让每个位置只能看到它之前的 token，看不到“答案”，这样才能用“预测下一个 token”来训练，也和生成时一次出一个 token 的方式一致。\n
\n\n
3. 多头注意力比单头好在哪？\n\n不同的头在不同的子空间里计算注意力，可以同时关注不同类型的关系（例如语法搭配和指代），再把结果拼起来。\n
\n\n
4. 上下文长度翻倍，注意力分数矩阵的计算量大约变成多少倍？\n\n大约 4 倍，因为它随序列长度的平方增长。\n
" } } }