How LLMs Generate Text: Decoding, Temperature, KV Cache and Quantisation

Two phases: prefill and decode

When you send a prompt, the inference engine does two different kinds of work:

1. Prefill – the whole prompt goes through the model in one parallel pass. This is compute-heavy but fast per token, and determines the time to first token.
2. Decode – the model produces the answer one token at a time: run the network, get probabilities for the next token, choose one, append it, repeat (see What Is an LLM?). This determines tokens per second.

sequenceDiagram
  participant U as User
  participant E as Inference engine
  participant M as Model weights
  U->>E: prompt (1,000 tokens)
  E->>M: Prefill: all prompt tokens in parallel
  M-->>E: keys/values for every layer → KV cache
  loop each new token
    E->>M: Decode: 1 new token + KV cache
    M-->>E: probabilities for next token
    E->>E: sample token, append K/V to cache
  end
  E-->>U: streamed answer

Choosing the next token: sampling

The model outputs a probability for every token in its vocabulary. How you pick from it matters:

• Greedy – always take the most likely token. Deterministic, but prone to repetition.
• Temperature – divide the scores (logits) by a temperature T before softmax. T < 1 sharpens the distribution (safer), *T* > 1 flattens it (more diverse).
• Top-k / top-p – sample only from the k most likely tokens, or from the smallest set whose probabilities add up to p (nucleus sampling, Holtzman et al., 2019). This cuts off the long tail of nonsense tokens.

For code, extraction and agents, low temperature is usual; for brainstorming and fiction, higher. Even at temperature 0, outputs from hosted APIs are not always bit-for-bit identical, because batching and floating-point ordering vary.

The KV cache

In attention, each new token needs the keys and values of all previous tokens (Transformers and Attention). Recomputing them every step would make generation quadratically slow, so engines keep them in a KV cache and only compute K and V for the new token.

The cache costs memory: 2 × layers × KV-heads × head-size × bytes × tokens. For Llama 3 8B (32 layers, 8 KV heads of size 128, 16-bit) that is about 128 KB per token — roughly 1 GB for an 8,000-token context, per conversation. Techniques such as grouped-query attention (Ainslie et al., 2023), which shares K/V across heads, exist largely to shrink this cache.

Why generation is memory-bound

To produce one token, the hardware must read all the weights from memory. An 8B model at 16-bit is 16 GB; at, say, 400 GB/s of memory bandwidth, that caps a single stream at around 25 tokens per second regardless of how many FLOPs the chip has. Hence the main speed levers:

Quantisation: big models, small machines

Training usually uses 16- or 32-bit numbers. Quantisation stores weights with fewer bits after training — e.g. LLM.int8() (Dettmers et al., 2022) and GPTQ (Frantar et al., 2022). The llama.cpp project popularised the GGUF file format with many quantisation types (names like Q4_K_M, Q8_0).

Rule-of-thumb weight sizes (excluding KV cache and runtime overhead):

Quality loss at 8-bit is usually negligible; good 4-bit quantisation is the common sweet spot for local use; below that, quality degrades faster.

Run it yourself

• Ollama and LM Studio download quantised models and show tokens per second.
• llama.cpp exposes every knob: quantisation type, context size, KV-cache type, batch size and speculative decoding.
• On Apple silicon, MLX LM runs and quantises models using unified memory.

Next steps: Inference Engines and Running LLMs on Your Own Computer (in progress), and GPUs, Memory and FLOPs for the hardware side.

Algorithm cards

Must-read papers

1. The Curious Case of Neural Text Degeneration (2019) — why greedy decoding repeats itself, and the proposal of top-p (nucleus) sampling. arXiv:1904.09751
2. Efficient Streaming Language Models with Attention Sinks (2023) — when the KV cache cannot grow forever, keeping "sink" tokens keeps streaming generation stable. arXiv:2309.17453

Hands-on project (about 6 hours)

Add a KV cache to a small GPT you trained (e.g. nanoGPT from the previous lesson), time 1,000 generated tokens with and without it, and compute the speed-up; then use the formula in self-check question 1 to estimate the KV cache of a 7B model at 32K context. Pair it with Lilian Weng's Large Transformer Model Inference Optimization; for sampling parameters see the Hugging Face generation strategies guide, and for how a production engine manages the KV cache read the vLLM docs.

Self-check