What Do LLMs Learn Inside? Representations, Features and Interpretability

Why look inside at all?

We know exactly how an LLM computes — every multiplication is specified. What we do not know is what the billions of learned numbers mean. Interpretability research tries to answer that, both out of scientific curiosity and because auditing and controlling powerful systems is easier if we can read what is going on inside them.

Recall the structure from Transformers and Attention: each token's vector flows through dozens of layers along a residual stream; attention layers move information between tokens and feed-forward (MLP) layers transform it. Interpretability asks what those intermediate vectors (activations) represent.

Finding 1: concepts are directions

A simple tool is the linear probe (Alain & Bengio, 2016): train a tiny linear classifier to read some property from a layer's activations. If it succeeds, the information is there in an easily usable form.

Probes keep finding rich structure:

• Board games: a GPT model trained only on sequences of legal Othello moves — never shown a board — internally represents the board state, and editing that representation changes which moves it predicts (Li et al., 2022).
• Space and time: Llama-2 has linear representations of the latitude/longitude of places and the dates of historical events and people (Gurnee & Tegmark, 2023).

This is the "linear representation" idea that started with word2vec (Tokens and Embeddings): many concepts correspond to directions in activation space.

Finding 2: where facts live

Feed-forward layers behave like key–value memories: some neurons fire on input patterns and push specific output tokens (Geva et al., 2020). ROME (Meng et al., 2022) traced factual recall ("The Eiffel Tower is in … Paris") to MLP layers in the middle of GPT models and showed the association could be edited — making the model "believe" the tower is in Rome — with a targeted weight change.

Finding 3: superposition — more concepts than neurons

If each neuron meant one thing, we could just read neurons. Usually they do not: a single neuron may respond to Korean text, citations and HTTP requests. Elhage et al. (2022) showed in toy models why: when there are more useful features than dimensions, and features are rarely active at the same time, networks store them in superposition — as overlapping, almost-orthogonal directions.

Finding 4: dictionary learning finds millions of features

Sparse autoencoders (SAEs) undo superposition. They learn to rewrite each activation as a sum of a few active "features" from a very large dictionary.

flowchart LR
  A["Activation vector (thousands of dims, polysemantic)"] --> E["SAE encoder"]
  E --> F["Millions of sparse features (only a few active)"]
  F --> D["SAE decoder"]
  D --> R["Reconstructed activation"]
  F --> L["Label features by the texts that activate them"]

Anthropic scaled this to a production model, extracting up to about 34 million features from Claude 3 Sonnet (Templeton et al., 2024); OpenAI trained a 16-million-feature SAE on GPT-4 activations (Gao et al., 2024). Features include concrete entities (the Golden Gate Bridge), abstract ideas (code bugs, sycophantic praise, secrecy) and are often multilingual and multimodal. Crucially, they are causal: clamping the Golden Gate Bridge feature high made the model bring the bridge into almost every answer.

Finding 5: circuits — how features combine

Features are the vocabulary; circuits are the algorithms.

• Induction heads (Olsson et al., 2022): a pair of attention heads that implements "if A B appeared earlier and I now see A, predict B" — a key mechanism behind in-context learning, which appears abruptly during training.
• Attribution graphs (Lindsey et al., 2025) trace which features cause an output in Claude 3.5 Haiku. Examples: answering "the capital of the state containing Dallas" goes internally through Texas before Austin (genuine two-step reasoning); when writing rhyming poetry the model picks the rhyme word before writing the line (planning ahead).

What we still don't know

• Current tools explain only a fraction of a model's computation; SAE reconstructions are imperfect, and labelling millions of features relies partly on automation.
• Self-explanations are not reliable. Chain-of-thought text can leave out the real cause of an answer (Turpin et al., 2023), and attribution-graph studies found cases where the model's described method differs from what it computed.
• Whether these pieces add up to a full "world model" — and how far the methods scale to the largest models — are open research questions.

Go deeper

Continue with Circuits and Sparse Autoencoders and Probing, Editing and Steering (in progress). The Transformer Circuits Thread (transformer-circuits.pub) and Google DeepMind's open Gemma Scope SAEs are good starting points for hands-on work.