// ◎ Transformers & Attention

Multi-head Latent Attention (MLA)

Caches a compressed low-rank latent instead of full keys and values.

多头潜在注意力 MLA 2024 Advanced core

IN PLAIN WORDS

Each token caches one small latent vector that is projected back to per-head keys/values at attention time — much smaller than GQA with MHA-like quality. Introduced in DeepSeek-V2, kept in V3/R1.

Used in: DeepSeek-V2/V3/R1, Kimi K2

TIMELINE · prerequisites → this → what builds on it

2023 Grouped-Query…2024 Multi-head Latent…

01Original paper

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

DeepSeek-AI, Aixin Liu, Bei Feng, et al. · 2024

arXiv ↗ ✓ Verified · arXiv API

02Where it sits

Learn the left column first; the right column builds on this one.

Learn first

Multi-head Latent Attention (MLA)

Then you can learn

Nothing in the atlas builds on it yet

Left: prerequisites · right: builds on this · dashed: related. Full map →

03Learning resources

Official docs, explainers, courses and reference code. All links checked.

Reference code

04Where the course teaches it

U42 Modern LLM Architecture RoPE, SwiGLU, RMSNorm, GQA/MLA, FlashAttention, sliding windows and context extension. 12 h12 algorithms5 papers

On learning paths:Understanding an LLM End to End

For AI / data

Verification: verified, checked 2026-10-21.
JSON: /api/learn/algorithms.json (id: mla)