Skip to content
English

Switch theme

Discover▾

Not sure where to start? This week’s hottest software, models and stacks in three minutes.

AI Apps▾

AI software people use: from assistants like ChatGPT to tools for coding, video and workflows, picked by purpose.

For Agents▾

Software for AI: tools that let your agent (Claude Code, Codex, Cursor…) browse, read docs, run code and remember.

Models▾

Choose models, estimate cost, self-host: prices, rankings, benchmarks, free tiers and compute.

Solutions▾

A goal or an error message? Ready-made stacks, steps and fixes by scenario.

Learn▾

A systematic course from zero to the frontier, with an algorithm library, must-read papers and tutorials.

Frontier▾

Daily AI news, weekly highlights, the latest papers and how AI is changing every discipline.

// Stage 6 · Inference, Deployment & Efficiency

Serving Engines

PagedAttention, continuous batching, prefix caching, speculative decoding, prefill/decode disaggregation and gateway routing.

What you will learn ↓ Code & exercises ↓

about 8 hours7 algorithms4 paperslesson text in progress

01Goals

Do these first: Decoding, Sampling & KV Cache

02Algorithms to master

Each card has the original paper, prerequisites and resources.

PagedAttention (vLLM) Manages KV cache in pages like OS virtual memory. 2023core Continuous (Iteration-Level) Batching Re-forms the batch every decoding step so finished requests leave immediately. 2022core RadixAttention (Prefix Caching) Reuses KV cache across requests sharing prefixes via a radix tree. 2023ext Speculative Decoding A small draft model proposes tokens that the big model verifies in one pass. 2022core Medusa Adds extra decoding heads so the model drafts its own speculative tokens. 2024ext EAGLE Speculates at the feature level for higher acceptance rates. 2024core Prefill–Decode Disaggregation Runs prompt prefill and token decoding on separate GPU pools. 2024ext

03Must-read papers

04Courses & videos

05Hands-on: project, code & exercises

Free, official or author-published pages. Most run in the browser or on a free Colab GPU.

PROJECT

Deploy an open model with vLLM, load-test throughput and first-token latency at different concurrency, then compare with speculative decoding on.

  1. 1
    docs.vllm.ai ↗Official docs

    Serve an OpenAI-compatible endpoint with vLLM and load-test throughput and TTFT.

  2. 2
    sgl-project/sglang ↗Reference code

    Run the same load test on SGLang and compare the effect of prefix caching.

06Self-test

Answer in your head, then open to check.

What is TTFT?

Time to first token.

Does speculative decoding change the output distribution?

Not when implemented with standard rejection sampling.

What core problem does PagedAttention solve?

Fragmentation and waste in KV-cache memory.

On these routes: Application Developers · Product Managers & Leaders

Data updated: AI algorithms & curriculum(UTC+8)