// ◎ Transformers & Attention
SentencePiece Tokenization
Trains subword tokenizers on raw text without relying on spaces – works for Chinese and other languages.
SentencePiece 分词 aka Unigram LM Tokenizer 2018 Foundational core
IN PLAIN WORDS
It treats the space as an ordinary symbol and learns a subword vocabulary straight from raw sentences, supporting both BPE and Unigram, with lossless detokenization. No language-specific pre-tokenization is needed, which suits languages without spaces such as Chinese and Japanese.
Used in: The tokenizer used by T5, Llama 1/2, Gemma and others
01Original paper
SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing
Taku Kudo, John Richardson · 2018
arXiv ↗ ✓ Verified · arXiv API
02Where it sits
Learn the left column first; the right column builds on this one.
Learn first
Then you can learn
- Nothing in the atlas builds on it yet
Related:T5 (Text-to-Text Transfer Transformer)
Left: prerequisites · right: builds on this · dashed: related. Full map →
03Learning resources
Official docs, explainers, courses and reference code. All links checked.
04Where the course teaches it
For AI / data
- Verification: verified, checked 2026-10-22.
- JSON: /api/learn/algorithms.json (id:
sentencepiece)