Skip to content
English

Switch theme

Discover▾

Not sure where to start? This week’s hottest software, models and stacks in three minutes.

AI Apps▾

AI software people use: from assistants like ChatGPT to tools for coding, video and workflows, picked by purpose.

For Agents▾

Software for AI: tools that let your agent (Claude Code, Codex, Cursor…) browse, read docs, run code and remember.

Models▾

Choose models, estimate cost, self-host: prices, rankings, benchmarks, free tiers and compute.

Solutions▾

A goal or an error message? Ready-made stacks, steps and fixes by scenario.

Learn▾

A systematic course from zero to the frontier, with an algorithm library, must-read papers and tutorials.

Frontier▾

Daily AI news, weekly highlights, the latest papers and how AI is changing every discipline.

// Stage 6 · Inference, Deployment & Efficiency

Quantization, Pruning & Distillation

INT8/INT4, GPTQ, AWQ, SmoothQuant, BitNet and knowledge distillation.

What you will learn ↓ Code & exercises ↓

about 8 hours7 algorithms4 paperslesson text in progress

01Goals

Do these first: Decoding, Sampling & KV Cache

02Algorithms to master

Each card has the original paper, prerequisites and resources.

LLM.int8() & Outlier Handling 8-bit matmul with outlier features kept in 16-bit. 2022core GPTQ Second-order, column-by-column post-training weight quantization to 3–4 bits. 2022core AWQ (Activation-aware Weight Quantization) Protects the ~1% most salient weights identified by activation magnitude. 2023core SmoothQuant Migrates quantization difficulty from activations to weights for W8A8. 2022core SparseGPT One-shot pruning of half a large model's weights without retraining. 2023ext BitNet b1.58 (Ternary LLMs) Trains LLMs from scratch with ternary weights {−1, 0, +1}. 2024core Knowledge Distillation A small student learns from a large teacher's soft probabilities. 2015core

03Must-read papers

04Courses & videos

05Hands-on: project, code & exercises

Free, official or author-published pages. Most run in the browser or on a free Colab GPU.

PROJECT

Quantise the same 7B model to 8, 4 and 3 bits; measure perplexity, speed and memory, and plot the trade-off.

  1. 1
    ggml-org/llama.cpp ↗Reference code

    Convert a model to GGUF, quantise to Q8/Q4 and compare speed and perplexity.

  2. 2
    huggingface.co ↗Official docs

    Load one model via bitsandbytes / GPTQ / AWQ and compare memory.

  3. 3
    hanlab.mit.edu ↗Courses

    Do the MIT 6.5940 labs (pruning, quantisation, distillation).

06Self-test

Answer in your head, then open to check.

Which is harder to quantise, weights or activations?

Activations, because of outliers.

Which weights does AWQ protect?

The salient weights tied to large-activation channels.

What information do “soft labels” carry in distillation?

The relative similarity between wrong classes.

On these routes: Algorithm Researchers

Data updated: AI algorithms & curriculum(UTC+8)