// ◈ Algorithms Atlas
Quantization, Pruning & Distillation
Making models smaller, faster and lighter so they run on one GPU, a laptop or a phone.
6 core1 extended2015–2024
TIMELINE
01Core algorithms
Learn these first. Sorted by year.
Knowledge Distillation A small student learns from a large teacher's soft probabilities. 2015core LLM.int8() & Outlier Handling 8-bit matmul with outlier features kept in 16-bit. 2022core GPTQ Second-order, column-by-column post-training weight quantization to 3–4 bits. 2022core SmoothQuant Migrates quantization difficulty from activations to weights for W8A8. 2022core AWQ (Activation-aware Weight Quantization) Protects the ~1% most salient weights identified by activation magnitude. 2023core BitNet b1.58 (Ternary LLMs) Trains LLMs from scratch with ternary weights {−1, 0, +1}. 2024core
02Extended
Variants and follow-ups worth knowing once the core is clear.