// ◈ Quantization, Pruning & Distillation
Knowledge Distillation
A small student learns from a large teacher's soft probabilities.
知识蒸馏 aka On-Policy Distillation / 在线蒸馏 / GKD 2015 Foundational core
IN PLAIN WORDS
A teacher’s soft outputs reveal which wrong answers are plausible; a student mimicking them approaches the teacher at a fraction of the size. Two common modern forms: training on teacher-generated data, and on-policy distillation, where the student generates its own answers and the teacher scores them token by token, so the student learns from its own mistakes.
Used in: DistilBERT, R1-distilled models, on-device models; on-policy distillation in reasoning post-training (e.g. CriticOPD, arXiv 2610.20572, Oct 2026)
TIMELINE · prerequisites → this → what builds on it
01Original paper
Distilling the Knowledge in a Neural Network
Geoffrey Hinton, Oriol Vinyals, Jeff Dean · 2015
arXiv ↗ ✓ Verified · arXiv API
02Where it sits
Learn the left column first; the right column builds on this one.
Learn first
Then you can learn
Related:DeepSeek-R1 Reasoning RL RecipeSelf-Instruct
Left: prerequisites · right: builds on this · dashed: related. Full map →
03Learning resources
Official docs, explainers, courses and reference code. All links checked.
Books
Explainers
Official docs
04Where the course teaches it
For AI / data
- Verification: verified, checked 2026-10-21.
- JSON: /api/learn/algorithms.json (id:
knowledge-distillation)