Transformers and Attention Explained: Q, K, V, Multi-Head and the Decoder Block

The problem attention solves

Take the sentence "The animal didn't cross the street because it was too tired." To understand it, a model must connect it to animal, not street. Change tired to wide and the answer flips. Meaning depends on other words, sometimes far away.

Earlier models (RNNs, LSTMs) read text one token at a time and squeezed everything into a running memory, which made long-range links hard and training slow. Attention was first added on top of such models for translation (Bahdanau et al., 2014); the Transformer (Vaswani et al., 2017) went further and built the whole model out of attention — hence the title Attention Is All You Need.

Self-attention in four steps

Each token starts as a vector (its embedding, see Tokens and Embeddings). For every token, the layer computes three new vectors by multiplying with learned matrices:

• Query (Q) – "what am I looking for?"
• Key (K) – "what do I contain?"
• Value (V) – "what information do I pass on if selected?"

Then:

1. Score: compare the token's query with every token's key using a dot product. A high score means "relevant".
2. Scale: divide by √dₖ (the square root of the key size) to keep scores in a range where training is stable.
3. Normalise: apply softmax so the scores become positive weights that sum to 1.
4. Mix: the token's output is the weighted sum of all value vectors.

In matrix form, for all tokens at once:

Attention(Q, K, V) = softmax(Q·Kᵀ / √dₖ) · V

flowchart LR
  X["Token vectors"] --> Q["Q = X·Wq"]
  X --> K["K = X·Wk"]
  X --> V["V = X·Wv"]
  Q --> S["Scores = Q·Kᵀ / √dk"]
  K --> S
  S --> M["Causal mask (hide future tokens)"]
  M --> SM["softmax → attention weights"]
  SM --> O["Weights · V = new token vectors"]
  V --> O

For it in our sentence, a well-trained head gives animal a large weight, so the new vector for it carries information about the animal.

Multi-head attention

One set of weights can only express one kind of relationship at a time. Multi-head attention runs several attention operations ("heads") in parallel, each with its own Q/K/V matrices, then concatenates their outputs. The original Transformer used 8 heads; large LLMs use dozens. Different heads end up tracking different things — previous token, syntactic subject, matching brackets, repeated phrases (What LLMs Learn Inside).

Where does word order come in?

Attention by itself treats the input as a set: shuffling tokens would not change the scores. Transformers therefore add positional information. The 2017 paper added fixed sine/cosine patterns to the embeddings; most current LLMs use rotary position embeddings (RoPE) (Su et al., 2021), which rotate Q and K vectors according to position so that scores depend on relative distance.

The decoder-only Transformer (GPT-style)

The original Transformer had an encoder (reads the source sentence) and a decoder (writes the translation). GPT models (Radford et al., 2019) and almost all chat LLMs keep only the decoder stack, with a causal mask: a token may attend only to itself and earlier tokens. That lets one pass over a training text produce a next-token prediction at every position at once — extremely efficient training.

One Transformer layer (block) is:

1. Normalisation → masked multi-head self-attention → add the input back (residual connection).
2. Normalisation → feed-forward network (two linear layers with a non-linearity, applied to each token separately) → residual connection.

flowchart TB
  IN["Token vectors (embeddings + position)"] --> N1["Norm"] --> A["Masked multi-head attention"] --> R1(("+"))
  IN --> R1
  R1 --> N2["Norm"] --> F["Feed-forward network"] --> R2(("+"))
  R1 --> R2
  R2 --> NEXT["… repeat for N layers …"] --> HEAD["Final norm → linear → softmax over vocabulary"]

Stack this block dozens of times (GPT-3's largest version had 96 layers), add a final layer that turns the last vector into probabilities over the vocabulary, and you have an LLM. The residual connections act as a shared "stream" that each layer reads from and writes into.

The cost of attention

Because every token is compared with every other token, the score matrix grows with the square of the sequence length. Doubling the context roughly quadruples that part of the work. Engineering advances — FlashAttention (Dao et al., 2022), grouped-query attention, the KV cache for generation (How LLMs Generate Text) — are why million-token context windows became practical.

See it run

Small open models run on a laptop with llama.cpp or Apple's MLX LM; their model cards list the number of layers, heads and the hidden size — the exact quantities described here. For visual walkthroughs, Jay Alammar's Illustrated Transformer and Harvard's Annotated Transformer (with code) are excellent next steps.

Algorithm cards

Must-read papers

1. Attention Is All You Need (2017) — introduces the Transformer; everything in this lesson comes from it. arXiv:1706.03762
2. BERT (2018) — the encoder-only, bidirectional route; the best contrast for "encoder vs decoder". arXiv:1810.04805
3. Language Models are Few-Shot Learners (GPT-3, 2020) — scaling a decoder-only Transformer yields in-context learning. arXiv:2005.14165

Hands-on project (about 16 hours)

Follow nanoGPT to train a character-level GPT on Shakespeare (or any small corpus), then plot one layer's attention weights as a heat map to see what different heads attend to. For the code walk-through use The Annotated Transformer and Karpathy's Let's reproduce GPT-2 (124M); for a full course, Stanford CS224N.

Self-check