// ⚡ Inference Acceleration & Serving
Continuous (Iteration-Level) Batching
Re-forms the batch every decoding step so finished requests leave immediately.
连续批处理 2022 Intermediate core
IN PLAIN WORDS
Static batches wait for the longest request. Iteration-level scheduling swaps in new requests as others finish, keeping the GPU busy.
Used in: vLLM, TGI, SGLang
01Original paper
Orca: A Distributed Serving System for Transformer-Based Generative Models
Yu et al. · OSDI 2022 · 2022
Open ↗ ✓ Verified · 官方页面 HTTP/WebFetch 核实
02Where it sits
Learn the left column first; the right column builds on this one.
Learn first
Continuous (Iteration-Level) Batching
Then you can learn
Related:PagedAttention (vLLM)
Left: prerequisites · right: builds on this · dashed: related. Full map →
03Learning resources
Official docs, explainers, courses and reference code. All links checked.
Official docs
Explainers
04Where the course teaches it
On learning paths:AI Application Engineer
For AI / data
- Verification: verified, checked 2026-10-21.
- JSON: /api/learn/algorithms.json (id:
continuous-batching)