// ◎ Transformers & Attention

Byte-Pair Encoding (BPE) Tokenization

Builds a subword vocabulary by repeatedly merging the most frequent pairs.

BPE 子词分词 2015 Beginner core

IN PLAIN WORDS

Start from characters or bytes and repeatedly merge the most frequent adjacent pair into a new token. It can spell any word without overly long sequences; GPT and Llama tokenizers build on it.

Used in: Tokenizers of GPT (tiktoken), Llama, Qwen

TIMELINE · prerequisites → this → what builds on it

2015 Byte-Pair Encodin…2018 SentencePiece…

01Original paper

Neural Machine Translation of Rare Words with Subword Units

Rico Sennrich, Barry Haddow, Alexandra Birch · 2015

arXiv ↗ ✓ Verified · arXiv API

02Where it sits

Learn the left column first; the right column builds on this one.

Learn first

No prerequisites — a good starting point

Byte-Pair Encoding (BPE) Tokenization

Then you can learn

SentencePiece Tokenization

Left: prerequisites · right: builds on this · dashed: related. Full map →

03Learning resources

Official docs, explainers, courses and reference code. All links checked.

Reference code

Courses

huggingface.co ↗EN

04Where the course teaches it

U40 Tokenization & Embeddings BPE tokenisation, per-token pricing, embeddings and semantic similarity. 6 h6 algorithms2 paperslesson ready

On learning paths:Understanding an LLM End to End

For AI / data

Verification: verified, checked 2026-10-21.
JSON: /api/learn/algorithms.json (id: bpe)