Transformers & LLM Principles
Tokens and Embeddings: How LLMs Turn Text into Numbers
Intermediate 15 min For: Beginners / Developers / Researchers
Key takeaways
- ▸LLMs read and write tokens — frequent words, word pieces, bytes — chosen by an algorithm such as byte-pair encoding (BPE).
- ▸Context windows, prices and speed are all measured in tokens; the same text can cost very different numbers of tokens in different languages.
- ▸Each token ID is mapped to an embedding: a vector of hundreds to thousands of numbers learned during training.
- ▸In embedding space, similar meanings end up close together — the basis of semantic search and RAG.
Why not just use letters or words?
A neural network computes with numbers, so text has to be turned into a sequence of IDs first. Two obvious options both fail:
- Whole words: the vocabulary would be millions of entries, and any new word, typo or name would be unknown.
- Single characters: sequences become very long, which makes models slow and makes it harder to learn meaning.
The compromise used by practically every LLM is subword tokenisation: frequent words get their own token, rarer words are split into pieces, and in the worst case text falls back to raw bytes, so nothing is ever unknown.
How BPE builds a vocabulary
Byte-pair encoding (BPE), adapted for neural networks by Sennrich et al. (2015), learns its vocabulary from data:
- Start with individual characters (or bytes) as the vocabulary.
- Count every adjacent pair in a large corpus; merge the most frequent pair into a new token (e.g.
t+h→th). - Repeat until the vocabulary reaches the target size.
GPT-2's tokeniser had a vocabulary of 50,257 tokens; Meta's Llama 3 uses about 128,000. SentencePiece is a popular library that applies BPE or a related method directly to raw text, which suits languages without spaces such as Chinese and Japanese.
flowchart LR T["'Tokenization is fun!'"] --> S["Token pieces: 'Token' 'ization' ' is' ' fun' '!'"] S --> I["Token IDs: e.g. 3404, 2065, 374, 2523, 0"] I --> E["Embedding lookup: one vector per ID"] E --> M["Transformer layers"]
(The IDs above are illustrative; each tokeniser has its own.)
Why token counts matter in practice
- Context window: a "128K context" model can attend to about 128,000 tokens — prompt, documents and answer combined.
- Cost and speed: APIs charge per input and output token, and generation speed is measured in tokens per second.
- Languages: tokenisers trained mostly on English need more tokens for the same content in many other languages, so the same paragraph can cost more. OpenAI suggests roughly 4 characters per token for English (help article); measure other languages with the actual tokeniser, for example tiktoken.
- Odd failures: models see
strawberryas a few tokens, not ten letters, which is one reason letter counting and some arithmetic are unreliable.
Embeddings: tokens become vectors
After tokenisation, each ID selects one row of an embedding matrix: a learned vector of hundreds to thousands of numbers (GPT-3's largest model used 12,288). At first these vectors are random. During training, backpropagation adjusts them like any other parameter, and tokens used in similar contexts drift towards similar vectors.
This idea is older than LLMs. word2vec (Mikolov et al., 2013) showed that word vectors learned from context capture relationships as directions — the famous example being that king − man + woman lands near queen. Inside an LLM, the initial token embedding is only the start: each Transformer layer rewrites it using the surrounding context, so the vector for "bank" ends up different in "river bank" and "bank account" (Transformers and Attention).
Embedding models and semantic search
A related but separate tool is the embedding model (e.g. Sentence-BERT and many commercial and open models). It maps a whole sentence or document to a single vector so that texts with similar meaning are close, typically measured by cosine similarity. That makes it possible to search by meaning instead of keywords — the foundation of vector databases and RAG.
Try it yourself
FAQ
How many words is one token?
For typical English text, OpenAI's rule of thumb is about 4 characters or ¾ of a word per token. For Chinese, Japanese and many other languages the ratio is different, so always count with the model's own tokenizer.
Why can't LLMs count the letters in a word?
Because they usually see a word as one or two tokens, not as individual letters. Spelling tasks require the model to have memorised what is inside each token, which it often has not done reliably.
What is the difference between token embeddings and embedding models?
Token embeddings are the first layer inside an LLM. An embedding model is a separate model that turns a whole sentence or document into one vector, used for search, clustering and RAG.