// Stage 6 · Inference, Deployment & Efficiency
Decoding, Sampling & KV Cache
Prefill and decode, temperature/top-p, KV cache and the memory-bandwidth bottleneck.
about 6 hours4 algorithms2 paperslesson text ready
01Goals
- Estimate a model’s KV cache size
- Explain why decoding is bandwidth-bound
- Tune sampling parameters
Do these first: Modern LLM Architecture
02Read on this site
03Algorithms to master
Each card has the original paper, prerequisites and resources.
04Must-read papers
- The Curious Case of Neural Text Degeneration ↗2019 ✓ verified
- Efficient Streaming Language Models with Attention Sinks ↗2023 ✓ verified
05Courses & videos
- KV Cache ↗Official docsEN
- Lilian Weng：Large Transformer Model Inference Optimization ↗ExplainersEN
- Hugging Face Transformers ↗Official docsEN
- docs.vllm.ai ↗Official docsEN
06Hands-on: project, code & exercises
Free, official or author-published pages. Most run in the browser or on a free Colab GPU.
PROJECT
Add a KV cache to your small GPT and measure the speed-up when generating 1,000 tokens.
- 1Hugging Face Transformers ↗Official docs
Compare greedy, beam, top-p and speculative decoding outputs and speed on one model.
- 2
07Self-test
Answer in your head, then open to check.
KV cache size is proportional to what?
Layers × KV heads × head dimension × sequence length × batch × 2.
What does top-p = 0.9 mean?
Sample only from the smallest set of candidates whose cumulative probability reaches 90%.
Why is GPU utilisation low during decoding?
Each step computes one token; most time goes to reading weights and the cache.
On these routes: Application Developers