{
  "id": "inside-llms",
  "module": "interpretability",
  "level": 3,
  "minutes": 25,
  "audiences": [
    "beginner",
    "developer",
    "researcher"
  ],
  "software": [],
  "text": {
    "en": {
      "title": "What Do LLMs Learn Inside? Representations, Features and Interpretability",
      "description": "What researchers have found inside large language models: linear representations, world models, factual memories in MLP layers, induction heads, superposition, sparse-autoencoder features and attribution graphs — and the open questions.",
      "updated": "2026-10-10",
      "takeaways": [
        "Concepts are often represented as directions in the model's activation space; simple linear probes can read out things like truth, location, time or a board-game state.",
        "Models pack many more concepts than they have neurons by using superposition, so individual neurons are hard to interpret.",
        "Sparse autoencoders split activations into millions of more interpretable features; turning a feature up or down measurably changes behaviour.",
        "Circuit studies find concrete mechanisms (induction heads, multi-step lookup, planning ahead), but they explain only part of any model's behaviour."
      ],
      "faq": [
        {
          "q": "What is mechanistic interpretability?",
          "a": "The research field that tries to reverse-engineer neural networks into understandable mechanisms — which internal features exist and how they combine to produce an output — rather than only measuring inputs and outputs."
        },
        {
          "q": "Do LLMs have a world model?",
          "a": "There is evidence of internal models for specific domains: a model trained only on Othello moves represents the board, and Llama-2 has linear representations of geographic location and historical time. Whether this adds up to a general world model is still debated."
        },
        {
          "q": "Does the chain of thought show how the model really reasons?",
          "a": "Not reliably. Studies show written reasoning can omit the real factors behind an answer, and interpretability work has found cases where the internal computation differs from the model's own explanation."
        },
        {
          "q": "Why does interpretability matter for safety?",
          "a": "If we can read what features are active — for example deception- or danger-related ones — we can audit models, detect problems that tests miss and potentially steer behaviour. Today the tools are promising but incomplete."
        }
      ],
      "sources": [
        {
          "title": "Alain & Bengio (2016), Understanding intermediate layers using linear classifier probes",
          "url": "https://arxiv.org/abs/1610.01644"
        },
        {
          "title": "Li et al. (2022), Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task (Othello-GPT)",
          "url": "https://arxiv.org/abs/2210.13382"
        },
        {
          "title": "Gurnee & Tegmark (2023), Language Models Represent Space and Time",
          "url": "https://arxiv.org/abs/2310.02207"
        },
        {
          "title": "Geva et al. (2020), Transformer Feed-Forward Layers Are Key-Value Memories",
          "url": "https://arxiv.org/abs/2012.14913"
        },
        {
          "title": "Meng et al. (2022), Locating and Editing Factual Associations in GPT (ROME)",
          "url": "https://arxiv.org/abs/2202.05262"
        },
        {
          "title": "Elhage et al. (2022), Toy Models of Superposition",
          "url": "https://transformer-circuits.pub/2022/toy_model/index.html"
        },
        {
          "title": "Olsson et al. (2022), In-context Learning and Induction Heads",
          "url": "https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html"
        },
        {
          "title": "Templeton et al. (2024), Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet",
          "url": "https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html"
        },
        {
          "title": "Gao et al. (2024), Scaling and evaluating sparse autoencoders (OpenAI)",
          "url": "https://arxiv.org/abs/2406.04093"
        },
        {
          "title": "Lindsey et al. (2025), On the Biology of a Large Language Model",
          "url": "https://transformer-circuits.pub/2025/attribution-graphs/biology.html"
        },
        {
          "title": "Turpin et al. (2023), Language Models Don't Always Say What They Think",
          "url": "https://arxiv.org/abs/2305.04388"
        }
      ],
      "url": "https://aimap.page/en/learn/interpretability/inside-llms/",
      "markdown": "## Why look inside at all?\n\nWe know exactly how an LLM computes — every multiplication is specified. What we do not know is *what the billions of learned numbers mean*. **Interpretability** research tries to answer that, both out of scientific curiosity and because auditing and controlling powerful systems is easier if we can read what is going on inside them.\n\nRecall the structure from [Transformers and Attention](/learn/transformer-attention): each token's vector flows through dozens of layers along a **residual stream**; attention layers move information between tokens and feed-forward (MLP) layers transform it. Interpretability asks what those intermediate vectors (*activations*) represent.\n\n## Finding 1: concepts are directions\n\nA simple tool is the **linear probe** ([Alain & Bengio, 2016](https://arxiv.org/abs/1610.01644)): train a tiny linear classifier to read some property from a layer's activations. If it succeeds, the information is there in an easily usable form.\n\nProbes keep finding rich structure:\n\n- **Board games**: a GPT model trained only on sequences of legal Othello moves — never shown a board — internally represents the board state, and editing that representation changes which moves it predicts ([Li et al., 2022](https://arxiv.org/abs/2210.13382)).\n- **Space and time**: Llama-2 has linear representations of the latitude/longitude of places and the dates of historical events and people ([Gurnee & Tegmark, 2023](https://arxiv.org/abs/2310.02207)).\n\nThis is the \"linear representation\" idea that started with word2vec ([Tokens and Embeddings](/learn/tokens-embeddings)): many concepts correspond to directions in activation space.\n\n## Finding 2: where facts live\n\nFeed-forward layers behave like **key–value memories**: some neurons fire on input patterns and push specific output tokens ([Geva et al., 2020](https://arxiv.org/abs/2012.14913)). **ROME** ([Meng et al., 2022](https://arxiv.org/abs/2202.05262)) traced factual recall (\"The Eiffel Tower is in … Paris\") to MLP layers in the middle of GPT models and showed the association could be edited — making the model \"believe\" the tower is in Rome — with a targeted weight change.\n\n## Finding 3: superposition — more concepts than neurons\n\nIf each neuron meant one thing, we could just read neurons. Usually they do not: a single neuron may respond to Korean text, citations and HTTP requests. [Elhage et al. (2022)](https://transformer-circuits.pub/2022/toy_model/index.html) showed in toy models why: when there are more useful features than dimensions, and features are rarely active at the same time, networks store them in **superposition** — as overlapping, almost-orthogonal directions.\n\n## Finding 4: dictionary learning finds millions of features\n\n**Sparse autoencoders (SAEs)** undo superposition. They learn to rewrite each activation as a sum of a few active \"features\" from a very large dictionary.\n\n```mermaid\nflowchart LR\n  A[\"Activation vector (thousands of dims, polysemantic)\"] --> E[\"SAE encoder\"]\n  E --> F[\"Millions of sparse features (only a few active)\"]\n  F --> D[\"SAE decoder\"]\n  D --> R[\"Reconstructed activation\"]\n  F --> L[\"Label features by the texts that activate them\"]\n```\n\nAnthropic scaled this to a production model, extracting up to about 34 million features from Claude 3 Sonnet ([Templeton et al., 2024](https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html)); OpenAI trained a 16-million-feature SAE on GPT-4 activations ([Gao et al., 2024](https://arxiv.org/abs/2406.04093)). Features include concrete entities (the Golden Gate Bridge), abstract ideas (code bugs, sycophantic praise, secrecy) and are often **multilingual and multimodal**. Crucially, they are causal: clamping the Golden Gate Bridge feature high made the model bring the bridge into almost every answer.\n\n## Finding 5: circuits — how features combine\n\nFeatures are the vocabulary; **circuits** are the algorithms.\n\n- **Induction heads** ([Olsson et al., 2022](https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html)): a pair of attention heads that implements \"if `A B` appeared earlier and I now see `A`, predict `B`\" — a key mechanism behind in-context learning, which appears abruptly during training.\n- **Attribution graphs** ([Lindsey et al., 2025](https://transformer-circuits.pub/2025/attribution-graphs/biology.html)) trace which features cause an output in Claude 3.5 Haiku. Examples: answering \"the capital of the state containing Dallas\" goes internally through *Texas* before *Austin* (genuine two-step reasoning); when writing rhyming poetry the model picks the rhyme word **before** writing the line (planning ahead).\n\n## What we still don't know\n\n- Current tools explain only a fraction of a model's computation; SAE reconstructions are imperfect, and labelling millions of features relies partly on automation.\n- **Self-explanations are not reliable.** Chain-of-thought text can leave out the real cause of an answer ([Turpin et al., 2023](https://arxiv.org/abs/2305.04388)), and attribution-graph studies found cases where the model's described method differs from what it computed.\n- Whether these pieces add up to a full \"world model\" — and how far the methods scale to the largest models — are open research questions.\n\n## Go deeper\n\nContinue with [Circuits and Sparse Autoencoders](/learn/circuits-sae) and [Probing, Editing and Steering](/learn/probing-steering) (in progress). The Transformer Circuits Thread (transformer-circuits.pub) and Google DeepMind's open Gemma Scope SAEs are good starting points for hands-on work."
    },
    "zh-hans": {
      "title": "大模型内部学到了什么？表征、特征与可解释性",
      "description": "研究者在大模型内部发现了什么：线性表征、世界模型、存放在 MLP 层的事实记忆、归纳头、叠加、稀疏自编码器特征和归因图，以及仍未解决的问题。",
      "updated": "2026-10-10",
      "takeaways": [
        "概念常常表现为模型激活空间里的方向；简单的线性探针就能读出真假、地理位置、时间，甚至棋盘状态。",
        "模型用“叠加”存放远多于神经元数量的概念，所以单个神经元很难解释。",
        "稀疏自编码器把激活拆成数百万个更好解释的特征；调高或调低某个特征，会明显改变模型行为。",
        "电路研究找到了具体机制（归纳头、多步推理、提前规划），但它们只解释了模型行为的一部分。"
      ],
      "faq": [
        {
          "q": "什么是机制可解释性？",
          "a": "这个研究方向试图把神经网络“逆向工程”成人能理解的机制：内部有哪些特征、它们如何组合产生输出，而不只是测量输入和输出。"
        },
        {
          "q": "大模型有世界模型吗？",
          "a": "在特定领域有证据：只用黑白棋走法训练的模型，内部表征了棋盘；Llama-2 内部有地理位置和历史时间的线性表征。至于这些能否构成通用的世界模型，学界仍有争论。"
        },
        {
          "q": "思维链能反映模型真正的推理过程吗？",
          "a": "不能完全相信。研究表明，写出来的推理可能漏掉影响答案的真实因素；可解释性研究也发现过模型内部的计算方式和它自己的解释不一致的情况。"
        },
        {
          "q": "可解释性对 AI 安全有什么用？",
          "a": "如果能读出哪些特征被激活（比如与欺骗或危险相关的特征），就能审计模型、发现测试发现不了的问题，甚至引导模型行为。目前这些工具很有前景，但还不完整。"
        }
      ],
      "sources": [
        {
          "title": "Alain & Bengio（2016），Understanding intermediate layers using linear classifier probes",
          "url": "https://arxiv.org/abs/1610.01644"
        },
        {
          "title": "Li 等（2022），Emergent World Representations（Othello-GPT）",
          "url": "https://arxiv.org/abs/2210.13382"
        },
        {
          "title": "Gurnee & Tegmark（2023），Language Models Represent Space and Time",
          "url": "https://arxiv.org/abs/2310.02207"
        },
        {
          "title": "Geva 等（2020），Transformer Feed-Forward Layers Are Key-Value Memories",
          "url": "https://arxiv.org/abs/2012.14913"
        },
        {
          "title": "Meng 等（2022），Locating and Editing Factual Associations in GPT（ROME）",
          "url": "https://arxiv.org/abs/2202.05262"
        },
        {
          "title": "Elhage 等（2022），Toy Models of Superposition",
          "url": "https://transformer-circuits.pub/2022/toy_model/index.html"
        },
        {
          "title": "Olsson 等（2022），In-context Learning and Induction Heads",
          "url": "https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html"
        },
        {
          "title": "Templeton 等（2024），Scaling Monosemanticity（Claude 3 Sonnet 特征提取）",
          "url": "https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html"
        },
        {
          "title": "Gao 等（2024），Scaling and evaluating sparse autoencoders（OpenAI）",
          "url": "https://arxiv.org/abs/2406.04093"
        },
        {
          "title": "Lindsey 等（2025），On the Biology of a Large Language Model",
          "url": "https://transformer-circuits.pub/2025/attribution-graphs/biology.html"
        },
        {
          "title": "Turpin 等（2023），Language Models Don't Always Say What They Think",
          "url": "https://arxiv.org/abs/2305.04388"
        }
      ],
      "url": "https://aimap.page/zh-hans/learn/interpretability/inside-llms/",
      "markdown": "## 为什么要往里看？\n\n大模型是怎么计算的，我们一清二楚，每一次乘法都写得明明白白。我们不知道的是：*这几十亿个学出来的数字代表什么*。**可解释性**研究就是要回答这个问题，一方面出于科学好奇，另一方面，如果能读懂模型内部在发生什么，审计和控制强大的系统就容易得多。\n\n回顾 [Transformer 与注意力机制](/learn/transformer-attention) 中的结构：每个 token 的向量沿着一条**残差流**穿过几十层；注意力层在 token 之间搬运信息，前馈（MLP）层对信息做变换。可解释性要问的是：这些中间向量（*激活*）到底表示什么。\n\n## 发现一：概念是方向\n\n一个简单的工具是**线性探针**（[Alain & Bengio，2016](https://arxiv.org/abs/1610.01644)）：训练一个很小的线性分类器，从某一层的激活里读出某种属性。如果读得出来，说明这条信息以容易使用的形式存在。\n\n探针一次次发现了丰富的结构：\n\n- **棋盘**：一个只用合法黑白棋走法序列训练、从没见过棋盘的 GPT 模型，内部表征了棋盘状态，修改这个表征会改变它预测的走法（[Li 等，2022](https://arxiv.org/abs/2210.13382)）。\n- **空间与时间**：Llama-2 内部有地点经纬度、历史事件和人物年代的线性表征（[Gurnee & Tegmark，2023](https://arxiv.org/abs/2310.02207)）。\n\n这就是从 word2vec 开始的“线性表征”思想（见 [Token 与 Embedding](/learn/tokens-embeddings)）：很多概念对应激活空间里的方向。\n\n## 发现二：事实存在哪里\n\n前馈层的行为类似**键值记忆**：有些神经元对特定输入模式响应，并推动特定的输出 token（[Geva 等，2020](https://arxiv.org/abs/2012.14913)）。**ROME**（[Meng 等，2022](https://arxiv.org/abs/2202.05262)）把事实回忆（“埃菲尔铁塔位于……巴黎”）定位到 GPT 中间层的 MLP，并证明可以通过有针对性地改动权重来编辑这条关联，让模型“相信”铁塔在罗马。\n\n## 发现三：叠加，概念比神经元多\n\n如果每个神经元只代表一件事，直接读神经元就行了。可惜通常不是：同一个神经元可能对韩文、文献引用和 HTTP 请求都有反应。[Elhage 等（2022）](https://transformer-circuits.pub/2022/toy_model/index.html) 用玩具模型解释了原因：当有用的特征多于维度、而且这些特征很少同时出现时，网络会把它们以**叠加**的方式存放，也就是一组互相重叠、几乎正交的方向。\n\n## 发现四：字典学习找出数百万个特征\n\n**稀疏自编码器（SAE）**用来“拆开”叠加。它学着把每个激活改写成一本超大“字典”里少数几个活跃“特征”的加和。\n\n```mermaid\nflowchart LR\n  A[\"激活向量（几千维，一词多义）\"] --> E[\"SAE 编码器\"]\n  E --> F[\"数百万个稀疏特征（每次只有少数活跃）\"]\n  F --> D[\"SAE 解码器\"]\n  D --> R[\"重建的激活\"]\n  F --> L[\"用激活它们的文本给特征命名\"]\n```\n\nAnthropic 把这种方法用到了生产级模型上，从 Claude 3 Sonnet 中提取了多达约 3400 万个特征（[Templeton 等，2024](https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html)）；OpenAI 在 GPT-4 的激活上训练了 1600 万特征的 SAE（[Gao 等，2024](https://arxiv.org/abs/2406.04093)）。特征既有具体实体（金门大桥），也有抽象概念（代码漏洞、谄媚式夸奖、保密），而且常常**跨语言、跨模态**。关键在于它们有因果作用：把“金门大桥”特征固定在高位，模型几乎每个回答都会扯到这座桥。\n\n## 发现五：电路，特征如何组合\n\n特征是词汇，**电路**是算法。\n\n- **归纳头**（[Olsson 等，2022](https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html)）：一对注意力头实现“如果前面出现过 `A B`，现在又看到 `A`，就预测 `B`”。这是上下文学习背后的关键机制，而且在训练中是突然出现的。\n- **归因图**（[Lindsey 等，2025](https://transformer-circuits.pub/2025/attribution-graphs/biology.html)）追踪 Claude 3.5 Haiku 里是哪些特征导致了某个输出。例如：回答“达拉斯所在州的首府是哪里”，模型内部会先经过“得克萨斯”再到“奥斯汀”，是真正的两步推理；写押韵诗时，模型会在写这一行**之前**先选好韵脚词，也就是提前规划。\n\n## 我们还不知道什么\n\n- 现有工具只能解释模型计算的一小部分；SAE 的重建并不完美，给数百万个特征命名也部分依赖自动化。\n- **模型的自我解释并不可靠。**思维链文本可能漏掉答案的真实原因（[Turpin 等，2023](https://arxiv.org/abs/2305.04388)）；归因图研究也发现过模型声称的方法和它实际计算不一致的例子。\n- 这些碎片能否拼成一个完整的“世界模型”，以及这些方法能扩展到多大的模型，都还是开放问题。\n\n## 继续深入\n\n接下来可以看 [电路与稀疏自编码器](/learn/circuits-sae) 和 [探针、知识编辑与激活引导](/learn/probing-steering)（编写中）。想动手的话，Transformer Circuits 系列文章（transformer-circuits.pub）和 Google DeepMind 开放的 Gemma Scope SAE 都是很好的起点。"
    }
  }
}