// ▲ Pretraining, Scaling & Distributed Training

Multi-Token Prediction (MTP)

Trains the model to predict several future tokens at once.

多 token 预测 2024 Advanced core

IN PLAIN WORDS

Extra heads on a shared trunk predict the 1st, 2nd, 3rd… next tokens. Denser training signal helps tasks like code, and the heads can drive speculative decoding.

Used in: DeepSeek-V3

TIMELINE · prerequisites → this → what builds on it

2017 Transformer &…2024 Multi-Token Predi…

01Original paper

Better & Faster Large Language Models via Multi-token Prediction

Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, et al. · 2024

arXiv ↗ ✓ Verified · arXiv API

02Where it sits

Learn the left column first; the right column builds on this one.

Learn first

Transformer & Self-Attention

Multi-Token Prediction (MTP)

Then you can learn

Nothing in the atlas builds on it yet

Related:Speculative Decoding Medusa

Left: prerequisites · right: builds on this · dashed: related. Full map →

03Learning resources

Official docs, explainers, courses and reference code. All links checked.

Reference code

deepseek-ai/DeepSeek-V3 ↗EN

04Where the course teaches it

U44 Pretraining, Data & Scaling Laws Pre-training data pipelines, scaling laws, Chinchilla, μP, MTP and training stability. 10 h6 algorithms4 papers

For AI / data

Verification: verified, checked 2026-10-21.
JSON: /api/learn/algorithms.json (id: mtp)