// ◎ Transformers & Attention
Grouped-Query Attention (GQA)
Groups heads to share K/V, a middle ground between MHA and MQA.
分组查询注意力 GQA 2023 Intermediate core
IN PLAIN WORDS
E.g., 32 query heads share 8 KV heads, cutting the KV cache 4× with almost no quality loss. The most common attention variant in open models.
Used in: Llama 2 70B, Llama 3, Mistral, Qwen
TIMELINE · prerequisites → this → what builds on it
01Original paper
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
Joshua Ainslie, James Lee-Thorp, Michiel de Jong, et al. · 2023
arXiv ↗ ✓ Verified · arXiv API
02Where it sits
Learn the left column first; the right column builds on this one.
Learn first
Then you can learn
Related:KV Cache
Left: prerequisites · right: builds on this · dashed: related. Full map →
03Learning resources
Official docs, explainers, courses and reference code. All links checked.
04Where the course teaches it
On learning paths:Understanding an LLM End to End
For AI / data
- Verification: verified, checked 2026-10-21.
- JSON: /api/learn/algorithms.json (id:
gqa)