{
  "id": "tokens-embeddings",
  "module": "llm-core",
  "level": 2,
  "minutes": 15,
  "audiences": [
    "beginner",
    "developer",
    "researcher"
  ],
  "software": [
    "ollama",
    "lm-studio"
  ],
  "text": {
    "en": {
      "title": "Tokens and Embeddings: How LLMs Turn Text into Numbers",
      "description": "What a token is, how byte-pair encoding (BPE) tokenisers work, why token counts matter for cost and context, and how embeddings place meaning in a vector space.",
      "updated": "2026-10-10",
      "takeaways": [
        "LLMs read and write tokens — frequent words, word pieces, bytes — chosen by an algorithm such as byte-pair encoding (BPE).",
        "Context windows, prices and speed are all measured in tokens; the same text can cost very different numbers of tokens in different languages.",
        "Each token ID is mapped to an embedding: a vector of hundreds to thousands of numbers learned during training.",
        "In embedding space, similar meanings end up close together — the basis of semantic search and RAG."
      ],
      "faq": [
        {
          "q": "How many words is one token?",
          "a": "For typical English text, OpenAI's rule of thumb is about 4 characters or ¾ of a word per token. For Chinese, Japanese and many other languages the ratio is different, so always count with the model's own tokenizer."
        },
        {
          "q": "Why can't LLMs count the letters in a word?",
          "a": "Because they usually see a word as one or two tokens, not as individual letters. Spelling tasks require the model to have memorised what is inside each token, which it often has not done reliably."
        },
        {
          "q": "What is the difference between token embeddings and embedding models?",
          "a": "Token embeddings are the first layer inside an LLM. An embedding model is a separate model that turns a whole sentence or document into one vector, used for search, clustering and RAG."
        }
      ],
      "sources": [
        {
          "title": "Sennrich, Haddow & Birch (2015), Neural Machine Translation of Rare Words with Subword Units (BPE)",
          "url": "https://arxiv.org/abs/1508.07909"
        },
        {
          "title": "Kudo & Richardson (2018), SentencePiece",
          "url": "https://arxiv.org/abs/1808.06226"
        },
        {
          "title": "Mikolov et al. (2013), Efficient Estimation of Word Representations in Vector Space (word2vec)",
          "url": "https://arxiv.org/abs/1301.3781"
        },
        {
          "title": "OpenAI Help Center, What are tokens and how to count them?",
          "url": "https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them"
        },
        {
          "title": "OpenAI, tiktoken (BPE tokeniser)",
          "url": "https://github.com/openai/tiktoken"
        },
        {
          "title": "Reimers & Gurevych (2019), Sentence-BERT",
          "url": "https://arxiv.org/abs/1908.10084"
        }
      ],
      "url": "https://aimap.page/en/learn/llm-core/tokens-embeddings/",
      "markdown": "## Why not just use letters or words?\n\nA neural network computes with numbers, so text has to be turned into a sequence of IDs first. Two obvious options both fail:\n\n- **Whole words**: the vocabulary would be millions of entries, and any new word, typo or name would be unknown.\n- **Single characters**: sequences become very long, which makes models slow and makes it harder to learn meaning.\n\nThe compromise used by practically every LLM is **subword tokenisation**: frequent words get their own token, rarer words are split into pieces, and in the worst case text falls back to raw bytes, so nothing is ever unknown.\n\n## How BPE builds a vocabulary\n\n**Byte-pair encoding (BPE)**, adapted for neural networks by [Sennrich et al. (2015)](https://arxiv.org/abs/1508.07909), learns its vocabulary from data:\n\n1. Start with individual characters (or bytes) as the vocabulary.\n2. Count every adjacent pair in a large corpus; merge the most frequent pair into a new token (e.g. `t` + `h` → `th`).\n3. Repeat until the vocabulary reaches the target size.\n\nGPT-2's tokeniser had a vocabulary of 50,257 tokens; Meta's Llama 3 uses about 128,000. [SentencePiece](https://arxiv.org/abs/1808.06226) is a popular library that applies BPE or a related method directly to raw text, which suits languages without spaces such as Chinese and Japanese.\n\n```mermaid\nflowchart LR\n  T[\"'Tokenization is fun!'\"] --> S[\"Token pieces: 'Token' 'ization' ' is' ' fun' '!'\"]\n  S --> I[\"Token IDs: e.g. 3404, 2065, 374, 2523, 0\"]\n  I --> E[\"Embedding lookup: one vector per ID\"]\n  E --> M[\"Transformer layers\"]\n```\n\n*(The IDs above are illustrative; each tokeniser has its own.)*\n\n## Why token counts matter in practice\n\n- **Context window**: a \"128K context\" model can attend to about 128,000 tokens — prompt, documents and answer combined.\n- **Cost and speed**: APIs charge per input and output token, and generation speed is measured in tokens per second.\n- **Languages**: tokenisers trained mostly on English need more tokens for the same content in many other languages, so the same paragraph can cost more. OpenAI suggests roughly 4 characters per token for English ([help article](https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them)); measure other languages with the actual tokeniser, for example [tiktoken](https://github.com/openai/tiktoken).\n- **Odd failures**: models see `strawberry` as a few tokens, not ten letters, which is one reason letter counting and some arithmetic are unreliable.\n\n## Embeddings: tokens become vectors\n\nAfter tokenisation, each ID selects one row of an **embedding matrix**: a learned vector of hundreds to thousands of numbers (GPT-3's largest model used 12,288). At first these vectors are random. During training, backpropagation adjusts them like any other parameter, and tokens used in similar contexts drift towards similar vectors.\n\nThis idea is older than LLMs. **word2vec** ([Mikolov et al., 2013](https://arxiv.org/abs/1301.3781)) showed that word vectors learned from context capture relationships as directions — the famous example being that *king − man + woman* lands near *queen*. Inside an LLM, the initial token embedding is only the start: each Transformer layer rewrites it using the surrounding context, so the vector for \"bank\" ends up different in \"river bank\" and \"bank account\" ([Transformers and Attention](/learn/transformer-attention)).\n\n## Embedding models and semantic search\n\nA related but separate tool is the **embedding model** (e.g. [Sentence-BERT](https://arxiv.org/abs/1908.10084) and many commercial and open models). It maps a whole sentence or document to a single vector so that texts with similar meaning are close, typically measured by cosine similarity. That makes it possible to search by meaning instead of keywords — the foundation of vector databases and [RAG](/learn/what-is-rag).\n\n## Try it yourself\n\n- Paste a sentence into a tokeniser viewer (OpenAI provides one) in English and in Chinese and compare the counts.\n- Run a local model with [Ollama](/apps/ollama) or [LM Studio](/apps/lm-studio): both show tokens per second, and Ollama can also serve embedding models, so you can compute the cosine similarity of two sentences yourself."
    },
    "zh-hans": {
      "title": "Token 与 Embedding：大模型如何把文字变成数字",
      "description": "Token 是什么、BPE 分词器怎么工作、为什么 token 数决定了费用和上下文长度，以及 Embedding 如何把语义放进向量空间。",
      "updated": "2026-10-10",
      "takeaways": [
        "大模型读写的单位是 token：常见词、词片段或字节，由 BPE 之类的算法决定。",
        "上下文窗口、价格和速度都按 token 计算；同样的内容，用不同语言写，token 数可能差很多。",
        "每个 token 编号对应一个 Embedding：由几百到上万个数字组成、在训练中学出来的向量。",
        "在 Embedding 空间里，意思相近的内容彼此靠近，这是语义搜索和 RAG 的基础。"
      ],
      "faq": [
        {
          "q": "一个 token 等于几个字？",
          "a": "OpenAI 给出的经验值是：英文大约 4 个字符、约 3/4 个单词对应一个 token。中文因分词器而异，常见情况是一到两个汉字一个 token，准确数字要用对应模型的分词器去数。"
        },
        {
          "q": "为什么大模型数不清一个单词里有几个字母？",
          "a": "因为它通常把一个单词看成一两个 token，而不是一个个字母。要拼写或数字母，模型得记住每个 token 里面是哪些字母，这一点它往往记得不牢。"
        },
        {
          "q": "大模型里的 token embedding 和“向量模型”有什么区别？",
          "a": "token embedding 是大模型内部的第一层；向量模型（embedding 模型）是另一个独立的模型，把整句话或整篇文档变成一个向量，用于搜索、聚类和 RAG。"
        }
      ],
      "sources": [
        {
          "title": "Sennrich、Haddow、Birch（2015），Neural Machine Translation of Rare Words with Subword Units（BPE）",
          "url": "https://arxiv.org/abs/1508.07909"
        },
        {
          "title": "Kudo & Richardson（2018），SentencePiece",
          "url": "https://arxiv.org/abs/1808.06226"
        },
        {
          "title": "Mikolov 等（2013），Efficient Estimation of Word Representations in Vector Space（word2vec）",
          "url": "https://arxiv.org/abs/1301.3781"
        },
        {
          "title": "OpenAI 帮助中心，What are tokens and how to count them?",
          "url": "https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them"
        },
        {
          "title": "OpenAI，tiktoken（BPE 分词器）",
          "url": "https://github.com/openai/tiktoken"
        },
        {
          "title": "Reimers & Gurevych（2019），Sentence-BERT",
          "url": "https://arxiv.org/abs/1908.10084"
        }
      ],
      "url": "https://aimap.page/zh-hans/learn/llm-core/tokens-embeddings/",
      "markdown": "## 为什么不直接用字母或整词？\n\n神经网络只能拿数字计算，所以文字要先变成一串编号。两个直观的方案都行不通：\n\n- **整词**：词表会有几百万项，遇到新词、错别字、人名就不认识。\n- **单个字符**：序列变得很长，模型又慢，也更难学到含义。\n\n几乎所有大模型采用的折中方案是**子词分词**：常见词单独成为一个 token，少见的词拆成几块，最坏情况下退回到原始字节，所以永远不会出现“不认识”的输入。\n\n## BPE 怎样建立词表\n\n**字节对编码（BPE）**由 [Sennrich 等（2015）](https://arxiv.org/abs/1508.07909) 引入神经网络，它从数据中学出词表：\n\n1. 先把单个字符（或字节）作为初始词表。\n2. 在大量语料里统计所有相邻的两两组合，把出现最多的一对合并成新 token（比如 `t` + `h` → `th`）。\n3. 不断重复，直到词表达到目标大小。\n\nGPT-2 的词表有 50,257 个 token，Meta 的 Llama 3 约 128,000 个。[SentencePiece](https://arxiv.org/abs/1808.06226) 是一个常用库，直接在原始文本上做 BPE 或类似算法，很适合中文、日文这种没有空格分词的语言。\n\n```mermaid\nflowchart LR\n  T[\"“大模型很有趣！”\"] --> S[\"切成 token：'大' '模型' '很' '有趣' '！'\"]\n  S --> I[\"token 编号：如 1023, 52144, 3001, 8812, 6\"]\n  I --> E[\"查 Embedding 表：每个编号一个向量\"]\n  E --> M[\"Transformer 各层\"]\n```\n\n*（上面的切分和编号仅为示意，每个分词器都不同。）*\n\n## token 数为什么重要\n\n- **上下文窗口**：“128K 上下文”的模型能同时看到约 128,000 个 token，提示词、文档和回答加在一起算。\n- **费用与速度**：API 按输入、输出 token 计费，生成速度用“每秒多少 token”衡量。\n- **语言差异**：主要用英文训练的分词器，处理很多其他语言时需要更多 token，同样一段话可能更贵。OpenAI 给出的英文经验值是约 4 个字符一个 token（[帮助文档](https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them)）；中文请用实际的分词器去数，比如 [tiktoken](https://github.com/openai/tiktoken)。\n- **奇怪的失误**：模型把 `strawberry` 看成几个 token，而不是十个字母，这是它数字母、做某些算术时不可靠的原因之一。\n\n## Embedding：token 变成向量\n\n分词之后，每个编号会从 **Embedding 矩阵**里取出一行：一个由几百到上万个数字组成的向量（GPT-3 最大的模型用的是 12,288 维）。一开始这些向量是随机的，训练中反向传播会像调整其他参数一样调整它们，于是在相似语境中出现的 token，向量也会变得相似。\n\n这个思想比大模型更早。**word2vec**（[Mikolov 等，2013](https://arxiv.org/abs/1301.3781)）就展示了：从上下文学到的词向量会把关系编码成方向，著名的例子是 *国王 − 男人 + 女人* 落在 *王后* 附近。在大模型内部，初始的 token embedding 只是起点：每个 Transformer 层都会根据上下文改写它，所以“苹果”在“吃苹果”和“苹果公司”里，最终的向量并不一样（见 [Transformer 与注意力机制](/learn/transformer-attention)）。\n\n## 向量模型与语义搜索\n\n另一个相关但独立的工具是**向量模型**（embedding 模型，例如 [Sentence-BERT](https://arxiv.org/abs/1908.10084)，以及许多商用和开源模型）。它把整句话或整篇文档映射成一个向量，让意思相近的文本彼此靠近，通常用余弦相似度衡量。这样就能按“意思”而不是按关键词搜索，这正是向量数据库和 [RAG](/learn/what-is-rag) 的基础。\n\n## 动手试试\n\n- 把同一句话分别用中文和英文放进分词器查看工具（OpenAI 提供在线版），对比 token 数。\n- 用 [Ollama](/apps/ollama) 或 [LM Studio](/apps/lm-studio) 运行本地模型：两者都会显示每秒 token 数；Ollama 还能运行向量模型，你可以自己算两句话的余弦相似度。"
    }
  }
}