{ "id": "llm-inference", "module": "inference", "level": 2, "minutes": 20, "audiences": [ "beginner", "developer", "researcher" ], "software": [ "ollama", "llama-cpp", "lm-studio", "mlx-lm" ], "text": { "en": { "title": "How LLMs Generate Text: Decoding, Temperature, KV Cache and Quantisation", "description": "What happens when you press Enter: prefill and decode phases, sampling with temperature and top-p, the KV cache, why generation is memory-bound, batching, speculative decoding, and how quantisation lets big models run on small machines.", "updated": "2026-10-22", "takeaways": [ "Inference has two phases: prefill (process the whole prompt in parallel) and decode (generate one token at a time).", "Temperature and top-p control how the next token is sampled from the model's probabilities — creativity versus consistency.", "The KV cache stores past keys and values so each new token does not recompute the whole prompt; it is also what eats memory on long contexts.", "Token generation is usually limited by memory bandwidth, so smaller weights (quantisation) and batching are the main levers for speed and cost." ], "faq": [ { "q": "What does temperature do in an LLM?", "a": "It rescales the model's probabilities before sampling. Low temperature (e.g. 0–0.3) makes the most likely tokens dominate, giving focused, repeatable output; higher values (0.8–1.2) flatten the distribution and give more varied output — and more mistakes." }, { "q": "How much memory do I need to run a 7B or 8B model?", "a": "The weights take about parameters × bytes per parameter: an 8B model needs roughly 16 GB at 16-bit and about 4–5 GB at 4-bit quantisation, plus extra for the KV cache and runtime. On a Mac the memory is shared with the system." }, { "q": "Does quantisation make a model worse?", "a": "Somewhat, depending on the method and bit width. 8-bit is usually close to lossless; good 4-bit methods keep most quality for chat use; below 4 bits quality drops faster, especially for small models." } ], "sources": [ { "title": "Holtzman et al. (2019), The Curious Case of Neural Text Degeneration (nucleus / top-p sampling)", "url": "https://arxiv.org/abs/1904.09751" }, { "title": "Kwon et al. (2023), Efficient Memory Management for LLM Serving with PagedAttention (vLLM)", "url": "https://arxiv.org/abs/2309.06180" }, { "title": "Yu et al. (2022), Orca: A Distributed Serving System for Transformer-Based Generative Models (continuous batching), OSDI", "url": "https://www.usenix.org/conference/osdi22/presentation/yu" }, { "title": "Leviathan, Kalman & Matias (2022), Fast Inference from Transformers via Speculative Decoding", "url": "https://arxiv.org/abs/2211.17192" }, { "title": "Ainslie et al. (2023), GQA: Training Generalized Multi-Query Transformer Models", "url": "https://arxiv.org/abs/2305.13245" }, { "title": "Frantar et al. (2022), GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers", "url": "https://arxiv.org/abs/2210.17323" }, { "title": "Dettmers et al. (2022), LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale", "url": "https://arxiv.org/abs/2208.07339" }, { "title": "llama.cpp (GGUF format and quantisation types)", "url": "https://github.com/ggml-org/llama.cpp" } ], "url": "https://aimap.page/en/learn/inference/llm-inference/", "markdown": "## Two phases: prefill and decode\n\nWhen you send a prompt, the inference engine does two different kinds of work:\n\n1. **Prefill** – the whole prompt goes through the model in one parallel pass. This is compute-heavy but fast per token, and determines the *time to first token*.\n2. **Decode** – the model produces the answer **one token at a time**: run the network, get probabilities for the next token, choose one, append it, repeat (see [What Is an LLM?](/learn/what-is-an-llm)). This determines *tokens per second*.\n\n```mermaid\nsequenceDiagram\n participant U as User\n participant E as Inference engine\n participant M as Model weights\n U->>E: prompt (1,000 tokens)\n E->>M: Prefill: all prompt tokens in parallel\n M-->>E: keys/values for every layer → KV cache\n loop each new token\n E->>M: Decode: 1 new token + KV cache\n M-->>E: probabilities for next token\n E->>E: sample token, append K/V to cache\n end\n E-->>U: streamed answer\n```\n\n## Choosing the next token: sampling\n\nThe model outputs a probability for every token in its vocabulary. How you pick from it matters:\n\n- **Greedy** – always take the most likely token. Deterministic, but prone to repetition.\n- **Temperature** – divide the scores (logits) by a temperature *T* before softmax. *T* < 1 sharpens the distribution (safer), *T* > 1 flattens it (more diverse).\n- **Top-k / top-p** – sample only from the *k* most likely tokens, or from the smallest set whose probabilities add up to *p* (**nucleus sampling**, [Holtzman et al., 2019](https://arxiv.org/abs/1904.09751)). This cuts off the long tail of nonsense tokens.\n\nFor code, extraction and agents, low temperature is usual; for brainstorming and fiction, higher. Even at temperature 0, outputs from hosted APIs are not always bit-for-bit identical, because batching and floating-point ordering vary.\n\n## The KV cache\n\nIn attention, each new token needs the **keys and values** of all previous tokens ([Transformers and Attention](/learn/transformer-attention)). Recomputing them every step would make generation quadratically slow, so engines keep them in a **KV cache** and only compute K and V for the new token.\n\nThe cache costs memory: `2 × layers × KV-heads × head-size × bytes × tokens`. For Llama 3 8B (32 layers, 8 KV heads of size 128, 16-bit) that is about **128 KB per token** — roughly 1 GB for an 8,000-token context, per conversation. Techniques such as **grouped-query attention** ([Ainslie et al., 2023](https://arxiv.org/abs/2305.13245)), which shares K/V across heads, exist largely to shrink this cache.\n\n## Why generation is memory-bound\n\nTo produce one token, the hardware must read **all the weights** from memory. An 8B model at 16-bit is 16 GB; at, say, 400 GB/s of memory bandwidth, that caps a single stream at around 25 tokens per second regardless of how many FLOPs the chip has. Hence the main speed levers:\n\n| Lever | What it does |\n|---|---|\n| **Quantisation** | Store weights in 8, 4 or fewer bits → fewer bytes to read per token |\n| **Batching** | Serve many users per weight read; *continuous batching* adds/removes requests every step ([Orca](https://www.usenix.org/conference/osdi22/presentation/yu)) |\n| **PagedAttention** | Manage the KV cache in pages like virtual memory, so more requests fit ([vLLM](https://arxiv.org/abs/2309.06180)) |\n| **Speculative decoding** | A small draft model proposes several tokens; the big model verifies them in one pass ([Leviathan et al., 2022](https://arxiv.org/abs/2211.17192)) |\n\n## Quantisation: big models, small machines\n\nTraining usually uses 16- or 32-bit numbers. **Quantisation** stores weights with fewer bits after training — e.g. LLM.int8() ([Dettmers et al., 2022](https://arxiv.org/abs/2208.07339)) and GPTQ ([Frantar et al., 2022](https://arxiv.org/abs/2210.17323)). The [llama.cpp](https://github.com/ggml-org/llama.cpp) project popularised the **GGUF** file format with many quantisation types (names like `Q4_K_M`, `Q8_0`).\n\nRule-of-thumb weight sizes (excluding KV cache and runtime overhead):\n\n| Model size | 16-bit | 8-bit | ~4-bit |\n|---|---|---|---|\n| 8B | ~16 GB | ~8 GB | ~4.5–5 GB |\n| 32B | ~64 GB | ~32 GB | ~18–20 GB |\n| 70B | ~140 GB | ~70 GB | ~40 GB |\n\nQuality loss at 8-bit is usually negligible; good 4-bit quantisation is the common sweet spot for local use; below that, quality degrades faster.\n\n## Run it yourself\n\n- [Ollama](/apps/ollama) and [LM Studio](/apps/lm-studio) download quantised models and show tokens per second.\n- [llama.cpp](/apps/llama-cpp) exposes every knob: quantisation type, context size, KV-cache type, batch size and speculative decoding.\n- On Apple silicon, [MLX LM](/apps/mlx-lm) runs and quantises models using unified memory.\n\nNext steps: [Inference Engines](/learn/serving-engines) and [Running LLMs on Your Own Computer](/learn/local-llm) (in progress), and [GPUs, Memory and FLOPs](/learn/gpu-basics) for the hardware side.\n\n## Algorithm cards\n\n| Algorithm | In one line | Original paper |\n|---|---|---|\n| Temperature & nucleus (top-p) sampling | Controls randomness by sampling only from the top-p probability mass | [Holtzman et al., 2019](https://arxiv.org/abs/1904.09751) |\n| KV cache | Stores past keys and values so each step doesn't recompute them | No single origin paper; see the [Transformers docs](https://huggingface.co/docs/transformers/main/en/kv_cache) |\n| Attention sinks (StreamingLLM) | Keeping a few initial sink tokens enables endless streaming generation | [Xiao et al., 2023](https://arxiv.org/abs/2309.17453) |\n| Grouped-query attention (GQA) | Heads in a group share one K/V, shrinking the KV cache | [Ainslie et al., 2023](https://arxiv.org/abs/2305.13245) |\n\n## Must-read papers\n\n1. **The Curious Case of Neural Text Degeneration** (2019) — why greedy decoding repeats itself, and the proposal of top-p (nucleus) sampling. [arXiv:1904.09751](https://arxiv.org/abs/1904.09751)\n2. **Efficient Streaming Language Models with Attention Sinks** (2023) — when the KV cache cannot grow forever, keeping \"sink\" tokens keeps streaming generation stable. [arXiv:2309.17453](https://arxiv.org/abs/2309.17453)\n\n## Hands-on project (about 6 hours)\n\nAdd a KV cache to a small GPT you trained (e.g. nanoGPT from the previous lesson), time 1,000 generated tokens with and without it, and compute the speed-up; then use the formula in self-check question 1 to estimate the KV cache of a 7B model at 32K context. Pair it with Lilian Weng's [Large Transformer Model Inference Optimization](https://lilianweng.github.io/posts/2023-01-10-inference-optimization/); for sampling parameters see the [Hugging Face generation strategies guide](https://huggingface.co/docs/transformers/generation_strategies), and for how a production engine manages the KV cache read the [vLLM docs](https://docs.vllm.ai/en/latest/).\n\n## Self-check\n\n
1. What is the KV cache size proportional to?\n\nLayers × KV heads × head dimension × sequence length × batch size × 2 (one for K, one for V), times bytes per number.\n
\n\n
2. What does top-p = 0.9 mean?\n\nSort candidate tokens by probability and sample only from the smallest set whose cumulative probability reaches 90%.\n
\n\n
3. Why is GPU compute under-used during decoding?\n\nEach step computes a single new token, so most time goes into reading weights and the KV cache from memory — it is bandwidth-bound, not compute-bound; batching spreads each weight read across many requests.\n
" }, "zh-hans": { "title": "大模型如何生成文本：解码、温度、KV Cache 与量化", "description": "按下回车之后发生了什么：预填充与解码两个阶段、温度与 top-p 采样、KV Cache、为什么生成速度受内存带宽限制、批处理、投机解码，以及量化如何让大模型跑在小机器上。", "updated": "2026-10-22", "takeaways": [ "推理分两个阶段：预填充（并行处理整段提示）和解码（一次生成一个 token）。", "温度和 top-p 决定如何从模型给出的概率中采样下一个 token，在创造性和稳定性之间取舍。", "KV Cache 保存之前的键和值，生成新 token 时不必重算整段提示；长上下文时吃掉大量内存的也是它。", "生成 token 通常受内存带宽限制，所以缩小权重（量化）和批处理是提速降本的主要手段。" ], "faq": [ { "q": "temperature（温度）参数有什么用？", "a": "它在采样前对模型的概率做缩放。低温度（如 0–0.3）让最可能的 token 占主导，输出集中、可复现；高温度（0.8–1.2）让分布更平，输出更多样，错误也更多。" }, { "q": "运行 7B、8B 模型需要多少内存？", "a": "权重约等于“参数量 × 每个参数的字节数”：8B 模型 16 位精度约 16 GB，4 位量化约 4–5 GB，另外还要给 KV Cache 和运行时留空间。Mac 上这部分内存和系统共用。" }, { "q": "量化会让模型变差吗？", "a": "会有一些影响，取决于方法和位数。8 位通常几乎无损；好的 4 位方法在聊天场景下能保住大部分质量；低于 4 位，质量下降会明显加快，小模型尤其如此。" } ], "sources": [ { "title": "Holtzman 等（2019），The Curious Case of Neural Text Degeneration（nucleus / top-p 采样）", "url": "https://arxiv.org/abs/1904.09751" }, { "title": "Kwon 等（2023），Efficient Memory Management for LLM Serving with PagedAttention（vLLM）", "url": "https://arxiv.org/abs/2309.06180" }, { "title": "Yu 等（2022），Orca: A Distributed Serving System for Transformer-Based Generative Models（连续批处理），OSDI", "url": "https://www.usenix.org/conference/osdi22/presentation/yu" }, { "title": "Leviathan、Kalman、Matias（2022），Fast Inference from Transformers via Speculative Decoding", "url": "https://arxiv.org/abs/2211.17192" }, { "title": "Ainslie 等（2023），GQA: Training Generalized Multi-Query Transformer Models", "url": "https://arxiv.org/abs/2305.13245" }, { "title": "Frantar 等（2022），GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers", "url": "https://arxiv.org/abs/2210.17323" }, { "title": "Dettmers 等（2022），LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale", "url": "https://arxiv.org/abs/2208.07339" }, { "title": "llama.cpp（GGUF 格式与量化类型）", "url": "https://github.com/ggml-org/llama.cpp" } ], "url": "https://aimap.page/zh-hans/learn/inference/llm-inference/", "markdown": "## 两个阶段：预填充与解码\n\n你发出提示后，推理引擎要做两种不同的工作：\n\n1. **预填充（prefill）**：整段提示一次性并行通过模型。计算量大，但摊到每个 token 很快，决定了*首个 token 的等待时间*。\n2. **解码（decode）**：模型**一次生成一个 token**：跑一遍网络，得到下一个 token 的概率，选一个，接上去，再重复（见 [大模型是什么](/learn/what-is-an-llm)）。这决定了*每秒生成多少 token*。\n\n```mermaid\nsequenceDiagram\n participant U as 用户\n participant E as 推理引擎\n participant M as 模型权重\n U->>E: 提示（1000 个 token）\n E->>M: 预填充：并行处理全部提示 token\n M-->>E: 每一层的键和值 → KV Cache\n loop 每个新 token\n E->>M: 解码：1 个新 token + KV Cache\n M-->>E: 下一个 token 的概率\n E->>E: 采样，并把新的 K/V 存进缓存\n end\n E-->>U: 流式输出回答\n```\n\n## 怎么选下一个 token：采样\n\n模型给词表里每个 token 都算出一个概率，怎么从中挑选很关键：\n\n- **贪心**：永远选最可能的那个。结果确定，但容易重复。\n- **温度**：在 softmax 之前把分数（logits）除以温度 *T*。*T* < 1 让分布更尖（更稳），*T* > 1 让分布更平（更多样）。\n- **top-k / top-p**：只在最可能的 *k* 个 token 里采样，或者在概率加起来达到 *p* 的最小集合里采样（**nucleus 采样**，[Holtzman 等，2019](https://arxiv.org/abs/1904.09751)），砍掉长尾里的胡言乱语。\n\n写代码、做信息抽取、跑 Agent 时通常用低温度；头脑风暴和写小说用高一些。即使温度设为 0，云端 API 的输出也不一定每次逐字相同，因为批处理和浮点运算顺序会变化。\n\n## KV Cache\n\n在注意力计算中，每个新 token 都需要前面所有 token 的**键和值**（见 [Transformer 与注意力机制](/learn/transformer-attention)）。每一步都重算会让生成速度按平方变慢，所以引擎把它们存进 **KV Cache**，只为新 token 计算 K 和 V。\n\n缓存要占内存：`2 × 层数 × KV 头数 × 每头维度 × 字节数 × token 数`。以 Llama 3 8B 为例（32 层、8 个 KV 头、每头 128 维、16 位），每个 token 约 **128 KB**，8000 个 token 的上下文约 1 GB，而且是每个对话各占一份。**分组查询注意力**（[Ainslie 等，2023](https://arxiv.org/abs/2305.13245)）让多个头共享 K/V，很大程度上就是为了压缩这块缓存。\n\n## 为什么生成速度受内存限制\n\n每生成一个 token，硬件都要把**全部权重**从内存读一遍。8B 模型 16 位精度是 16 GB；如果内存带宽是 400 GB/s，单路生成速度上限就在每秒 25 个 token 左右，跟芯片算力有多强关系不大。所以主要的提速手段是：\n\n| 手段 | 作用 |\n|---|---|\n| **量化** | 用 8 位、4 位或更少位存权重，每个 token 要读的字节更少 |\n| **批处理** | 读一次权重服务多个用户；*连续批处理*每一步都能加入或移出请求（[Orca](https://www.usenix.org/conference/osdi22/presentation/yu)） |\n| **PagedAttention** | 像虚拟内存一样分页管理 KV Cache，同时容纳更多请求（[vLLM](https://arxiv.org/abs/2309.06180)） |\n| **投机解码** | 小的草稿模型先猜几个 token，大模型一次性验证（[Leviathan 等，2022](https://arxiv.org/abs/2211.17192)） |\n\n## 量化：让大模型跑在小机器上\n\n训练通常用 16 位或 32 位数字。**量化**在训练后用更少的位数存权重，例如 LLM.int8()（[Dettmers 等，2022](https://arxiv.org/abs/2208.07339)）和 GPTQ（[Frantar 等，2022](https://arxiv.org/abs/2210.17323)）。[llama.cpp](https://github.com/ggml-org/llama.cpp) 项目推广了 **GGUF** 文件格式和多种量化类型（名字像 `Q4_K_M`、`Q8_0`）。\n\n权重大小的经验值（不含 KV Cache 和运行时开销）：\n\n| 模型规模 | 16 位 | 8 位 | 约 4 位 |\n|---|---|---|---|\n| 8B | 约 16 GB | 约 8 GB | 约 4.5–5 GB |\n| 32B | 约 64 GB | 约 32 GB | 约 18–20 GB |\n| 70B | 约 140 GB | 约 70 GB | 约 40 GB |\n\n8 位的质量损失通常可以忽略；好的 4 位量化是本地使用最常见的平衡点；再往下，质量下降会加快。\n\n## 自己动手\n\n- [Ollama](/apps/ollama) 和 [LM Studio](/apps/lm-studio) 会下载量化好的模型，并显示每秒 token 数。\n- [llama.cpp](/apps/llama-cpp) 开放了所有参数：量化类型、上下文长度、KV Cache 类型、批大小和投机解码。\n- 在苹果芯片上，[MLX LM](/apps/mlx-lm) 利用统一内存运行和量化模型。\n\n下一步：[推理引擎](/learn/serving-engines) 和 [在自己电脑上运行大模型](/learn/local-llm)（编写中），硬件方面见 [GPU、显存与算力](/learn/gpu-basics)。\n\n## 算法卡片\n\n| 算法 | 一句话 | 原始论文 |\n|---|---|---|\n| 温度与 Top-p 采样 | 控制生成随机性：只从累计概率达到 p 的候选里抽样 | [Holtzman 等，2019](https://arxiv.org/abs/1904.09751) |\n| KV Cache | 把已生成 token 的 Key/Value 存起来，避免每步重复计算 | 无单一原始论文，见 [Transformers 文档](https://huggingface.co/docs/transformers/main/en/kv_cache) |\n| 注意力汇聚点 StreamingLLM | 保留开头几个“汇聚”token 的 KV，就能无限长流式生成 | [Xiao 等，2023](https://arxiv.org/abs/2309.17453) |\n| 分组查询注意力 GQA | 每组注意力头共享一份 K/V，显著缩小 KV Cache | [Ainslie 等，2023](https://arxiv.org/abs/2305.13245) |\n\n## 必读论文\n\n1. **The Curious Case of Neural Text Degeneration**（2019）——解释为什么贪心解码会重复、提出 top-p（核采样）。[arXiv:1904.09751](https://arxiv.org/abs/1904.09751)\n2. **Efficient Streaming Language Models with Attention Sinks**（2023）——KV Cache 不能无限长时，保留“汇聚点”token 就能稳定流式生成。[arXiv:2309.17453](https://arxiv.org/abs/2309.17453)\n\n## 动手项目（约 6 学时）\n\n给自己训练的小 GPT（例如上一课的 nanoGPT）加上 KV Cache，分别测量生成 1000 个 token 的耗时，算出提速倍数；再按下面自测第 1 题的公式估算一个 7B 模型在 32K 上下文下的 KV Cache 大小。推荐配合阅读 Lilian Weng 的 [Large Transformer Model Inference Optimization](https://lilianweng.github.io/posts/2023-01-10-inference-optimization/)；调采样参数可对照 [Hugging Face 生成策略文档](https://huggingface.co/docs/transformers/generation_strategies)，看工业级推理引擎怎么管理 KV Cache 可读 [vLLM 文档](https://docs.vllm.ai/en/latest/)。\n\n## 自测\n\n
1. KV Cache 的大小和哪些量成正比？\n\n层数 × KV 头数 × 每头维度 × 序列长度 × batch 大小 × 2（K 和 V 各一份），再乘以每个数的字节数。\n
\n\n
2. top-p = 0.9 是什么意思？\n\n把候选 token 按概率从高到低排，只在累计概率刚好达到 90% 的那一小撮里抽样。\n
\n\n
3. 为什么解码阶段 GPU 算力利用率低？\n\n每一步只算一个新 token，大部分时间花在从显存读权重和 KV Cache 上，受内存带宽限制而不是算力限制；批处理能把读一次权重的成本分摊给多个请求。\n
" } } }