{
  "id": "neural-networks",
  "module": "deep-learning",
  "level": 2,
  "minutes": 20,
  "audiences": [
    "beginner",
    "developer",
    "researcher"
  ],
  "software": [
    "uv"
  ],
  "text": {
    "en": {
      "title": "Neural Networks and Backpropagation, Explained Simply",
      "description": "What a neuron, a layer and an activation function are, why depth matters, and how backpropagation computes gradients for millions of weights — with a diagram and sources.",
      "updated": "2026-10-10",
      "takeaways": [
        "A neuron computes a weighted sum of its inputs plus a bias, then applies a non-linear activation function.",
        "Stacking layers of neurons lets a network build complex features out of simple ones; without non-linearity, any stack collapses into one linear function.",
        "Backpropagation applies the chain rule from the output back to the input to get the gradient for every weight in one backward pass.",
        "Modern LLMs are deep neural networks of this kind, with special layers (attention) and tricks (residual connections, normalisation) that make very deep stacks trainable."
      ],
      "faq": [
        {
          "q": "Is a neural network like the human brain?",
          "a": "Only loosely. Artificial neurons were inspired by biological ones, but they are simple maths functions, trained with backpropagation, which the brain is not known to use in the same way."
        },
        {
          "q": "What is an activation function?",
          "a": "A simple non-linear function applied after each weighted sum, such as ReLU (max(0, x)) or GELU. It is what lets networks represent curved, complex relationships."
        },
        {
          "q": "What is the difference between deep learning and machine learning?",
          "a": "Deep learning is the part of machine learning that uses neural networks with many layers. It learns its own features from raw data instead of relying on hand-designed ones."
        }
      ],
      "sources": [
        {
          "title": "Rumelhart, Hinton & Williams (1986), Learning representations by back-propagating errors, Nature",
          "url": "https://www.nature.com/articles/323533a0"
        },
        {
          "title": "Goodfellow, Bengio & Courville, Deep Learning, chapter 6 (Deep Feedforward Networks)",
          "url": "https://www.deeplearningbook.org/contents/mlp.html"
        },
        {
          "title": "3Blue1Brown, Neural Networks (video series)",
          "url": "https://www.3blue1brown.com/topics/neural-networks"
        },
        {
          "title": "Andrej Karpathy, micrograd: a tiny autograd engine",
          "url": "https://github.com/karpathy/micrograd"
        },
        {
          "title": "Krizhevsky, Sutskever & Hinton (2012), ImageNet Classification with Deep Convolutional Neural Networks",
          "url": "https://papers.nips.cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html"
        },
        {
          "title": "He et al. (2015), Deep Residual Learning for Image Recognition",
          "url": "https://arxiv.org/abs/1512.03385"
        }
      ],
      "url": "https://aimap.page/en/learn/deep-learning/neural-networks/",
      "markdown": "## One neuron\n\nAn artificial **neuron** does two things:\n\n1. It multiplies each input by a **weight**, adds them up and adds a **bias**: `z = w₁x₁ + w₂x₂ + … + b`.\n2. It passes `z` through an **activation function**, for example **ReLU**: `max(0, z)`.\n\nThe weights and bias are the parameters that training adjusts ([How Machines Learn](/learn/ml-basics)). On its own a neuron is a very simple decision rule. The power comes from connecting many.\n\n## Layers and depth\n\nNeurons are arranged in **layers**. Every neuron in a layer reads the outputs of the previous layer. A network with an input layer, one or more **hidden layers** and an output layer is a *multilayer perceptron*; \"deep\" learning simply means many hidden layers.\n\n```mermaid\nflowchart LR\n  subgraph Input\n    x1((x1)); x2((x2)); x3((x3))\n  end\n  subgraph Hidden1[\"Hidden layer 1\"]\n    h1((h)); h2((h)); h3((h)); h4((h))\n  end\n  subgraph Hidden2[\"Hidden layer 2\"]\n    g1((h)); g2((h)); g3((h))\n  end\n  subgraph Output\n    y((\"ŷ\"))\n  end\n  x1 --> h1 & h2 & h3 & h4\n  x2 --> h1 & h2 & h3 & h4\n  x3 --> h1 & h2 & h3 & h4\n  h1 & h2 & h3 & h4 --> g1 & g2 & g3\n  g1 & g2 & g3 --> y\n```\n\nWhy depth helps: each layer can build features out of the previous layer's features. In image networks, early layers detect edges, middle layers textures and parts, later layers whole objects. In language models, early layers handle word pieces and syntax, later layers more abstract meaning ([What LLMs Learn Inside](/learn/inside-llms)).\n\nWhy the activation function matters: a stack of purely linear layers is still just one linear function — depth would add nothing. The non-linearity is what lets networks approximate curved, complex relationships. Classic results show that networks with enough hidden units can approximate any reasonable function; in practice, *deep* networks do so far more efficiently than very wide shallow ones.\n\n## Backpropagation: blame assignment\n\nTraining needs the gradient of the loss with respect to **every** weight. Computing each one separately would be hopelessly slow for millions of weights. **Backpropagation**, popularised by [Rumelhart, Hinton and Williams (1986)](https://www.nature.com/articles/323533a0), does it in one backward sweep:\n\n1. **Forward pass**: compute the output and the loss, remembering intermediate values.\n2. **Backward pass**: starting from the loss, apply the calculus **chain rule** layer by layer. Each layer receives \"how much the loss changes if my output changes\" from the layer above and turns it into (a) gradients for its own weights and (b) the same signal for the layer below.\n3. **Update** all weights with gradient descent.\n\n```mermaid\nflowchart LR\n  I[\"Input\"] -->|\"forward\"| L1[\"Layer 1\"] -->|\"forward\"| L2[\"Layer 2\"] -->|\"forward\"| O[\"Output → loss\"]\n  O -.->|\"∂loss/∂out\"| L2 -.->|\"backward\"| L1\n```\n\nIntuitively, backprop is *blame assignment*: it tells each weight how much it contributed to the error. The backward pass costs roughly as much as a couple of forward passes, which is why training networks with billions of weights is feasible at all. Frameworks such as PyTorch and JAX do this automatically (\"autograd\"); Karpathy's [micrograd](https://github.com/karpathy/micrograd) implements it in about a hundred lines and is the best way to see that there is no magic.\n\n## What made deep networks work\n\nNeural networks are old; deep learning took off around 2012, when [AlexNet](https://papers.nips.cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html) won the ImageNet competition by a wide margin. Three things came together:\n\n- **Data**: large labelled datasets (ImageNet) and later the whole web.\n- **Compute**: GPUs, which do the required matrix multiplications in parallel ([GPUs, Memory and FLOPs](/learn/gpu-basics)).\n- **Tricks for very deep stacks**: ReLU activations, better initialisation, **residual connections** ([He et al., 2015](https://arxiv.org/abs/1512.03385)) that let signals skip layers, and **normalisation** layers. Every Transformer layer in an LLM uses residual connections and normalisation for exactly this reason.\n\n## From here to LLMs\n\nA large language model is a deep neural network whose layers alternate between **attention** (mixing information across tokens) and **feed-forward** blocks (the neuron layers described above, applied to each token). It is trained by backpropagation on the next-token loss. Continue with [Tokens and Embeddings](/learn/tokens-embeddings) and [Transformers and Attention](/learn/transformer-attention)."
    },
    "zh-hans": {
      "title": "神经网络与反向传播：通俗讲解",
      "description": "神经元、层、激活函数分别是什么，为什么“深”很重要，反向传播如何一次算出数百万个权重的梯度。附图示与出处。",
      "updated": "2026-10-10",
      "takeaways": [
        "一个神经元做的事：把输入加权求和、加上偏置，再经过一个非线性激活函数。",
        "把神经元一层层叠起来，网络就能用简单特征组合出复杂特征；没有非线性，再多层也等价于一个线性函数。",
        "反向传播从输出往输入方向套用链式法则，一次反向计算就得到每个权重的梯度。",
        "现代大模型就是这种深度神经网络，只是加了注意力层，以及让超深网络能训练的残差连接和归一化。"
      ],
      "faq": [
        {
          "q": "神经网络和人脑一样吗？",
          "a": "只是粗略相似。人工神经元受生物神经元启发，但本质是简单的数学函数，用反向传播训练；目前没有证据表明大脑以同样方式学习。"
        },
        {
          "q": "激活函数是什么？",
          "a": "在每次加权求和之后套用的一个简单非线性函数，比如 ReLU（max(0, x)）或 GELU。正是它让网络能表示弯曲、复杂的关系。"
        },
        {
          "q": "深度学习和机器学习有什么区别？",
          "a": "深度学习是机器学习里使用多层神经网络的那一部分。它直接从原始数据中自己学特征，而不是依赖人工设计的特征。"
        }
      ],
      "sources": [
        {
          "title": "Rumelhart、Hinton、Williams（1986），Learning representations by back-propagating errors，Nature",
          "url": "https://www.nature.com/articles/323533a0"
        },
        {
          "title": "Goodfellow 等，《深度学习》第 6 章（深度前馈网络）",
          "url": "https://www.deeplearningbook.org/contents/mlp.html"
        },
        {
          "title": "3Blue1Brown，神经网络系列视频",
          "url": "https://www.3blue1brown.com/topics/neural-networks"
        },
        {
          "title": "Andrej Karpathy，micrograd：一个极简自动求导引擎",
          "url": "https://github.com/karpathy/micrograd"
        },
        {
          "title": "Krizhevsky、Sutskever、Hinton（2012），ImageNet Classification with Deep Convolutional Neural Networks（AlexNet）",
          "url": "https://papers.nips.cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html"
        },
        {
          "title": "He 等（2015），Deep Residual Learning for Image Recognition（ResNet）",
          "url": "https://arxiv.org/abs/1512.03385"
        }
      ],
      "url": "https://aimap.page/zh-hans/learn/deep-learning/neural-networks/",
      "markdown": "## 一个神经元\n\n人工**神经元**只做两件事：\n\n1. 把每个输入乘以一个**权重**，加起来，再加上一个**偏置**：`z = w₁x₁ + w₂x₂ + … + b`。\n2. 把 `z` 送进**激活函数**，例如 **ReLU**：`max(0, z)`。\n\n权重和偏置就是训练要调整的参数（见 [机器是怎么“学习”的](/learn/ml-basics)）。单个神经元只是一条非常简单的判断规则，威力来自把很多神经元连起来。\n\n## 层与深度\n\n神经元按**层**排列，每一层的神经元读取上一层的输出。由输入层、一个或多个**隐藏层**、输出层组成的网络叫*多层感知机*；“深度”学习，就是隐藏层很多。\n\n```mermaid\nflowchart LR\n  subgraph IN[\"输入层\"]\n    x1((x1)); x2((x2)); x3((x3))\n  end\n  subgraph H1[\"隐藏层 1\"]\n    h1((h)); h2((h)); h3((h)); h4((h))\n  end\n  subgraph H2[\"隐藏层 2\"]\n    g1((h)); g2((h)); g3((h))\n  end\n  subgraph OUT[\"输出层\"]\n    y((\"ŷ\"))\n  end\n  x1 --> h1 & h2 & h3 & h4\n  x2 --> h1 & h2 & h3 & h4\n  x3 --> h1 & h2 & h3 & h4\n  h1 & h2 & h3 & h4 --> g1 & g2 & g3\n  g1 & g2 & g3 --> y\n```\n\n深度为什么有用：每一层都能用上一层的特征组合出新特征。在图像网络里，浅层识别边缘，中间层识别纹理和部件，深层识别整个物体。在语言模型里，浅层处理词片段和语法，深层处理更抽象的含义（见 [大模型内部学到了什么](/learn/inside-llms)）。\n\n激活函数为什么重要：纯线性层叠多少层，结果仍然只是一个线性函数，深度毫无意义。非线性让网络能逼近弯曲、复杂的关系。经典理论表明，隐藏单元足够多的网络可以逼近任何合理的函数；实践中，*深*网络比又宽又浅的网络效率高得多。\n\n## 反向传播：分配“责任”\n\n训练需要损失对**每一个**权重的梯度。对几百万个权重逐个计算，慢到不可能。**反向传播**（由 [Rumelhart、Hinton、Williams 在 1986 年](https://www.nature.com/articles/323533a0)推广）一次反向扫描就能全部算出：\n\n1. **前向传播**：算出输出和损失，并记住中间结果。\n2. **反向传播**：从损失出发，逐层套用微积分的**链式法则**。每一层从上一层收到“我的输出变一点，损失会变多少”，再把它变成两样东西：自己权重的梯度，以及传给下一层的同样信号。\n3. **更新**：用梯度下降更新所有权重。\n\n```mermaid\nflowchart LR\n  I[\"输入\"] -->|\"前向\"| L1[\"第 1 层\"] -->|\"前向\"| L2[\"第 2 层\"] -->|\"前向\"| O[\"输出 → 损失\"]\n  O -.->|\"∂损失/∂输出\"| L2 -.->|\"反向\"| L1\n```\n\n直观地说，反向传播就是*分配责任*：告诉每个权重它对误差贡献了多少。反向计算的开销大约相当于几次前向计算，这正是训练几十亿权重的网络成为可能的原因。PyTorch、JAX 等框架会自动完成这一步（自动求导）；Karpathy 的 [micrograd](https://github.com/karpathy/micrograd) 用一百来行代码实现了它，是理解“这里没有魔法”的最好材料。\n\n## 深度网络为什么后来才成功\n\n神经网络并不新，深度学习在 2012 年前后爆发，标志是 [AlexNet](https://papers.nips.cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html) 在 ImageNet 比赛中大幅领先。三件事同时到位：\n\n- **数据**：大型标注数据集（ImageNet），后来是整个互联网。\n- **算力**：GPU 能并行完成所需的矩阵乘法（见 [GPU、显存与算力](/learn/gpu-basics)）。\n- **训练超深网络的技巧**：ReLU 激活、更好的初始化、让信号可以跳过若干层的**残差连接**（[He 等，2015](https://arxiv.org/abs/1512.03385)），以及**归一化**层。大模型的每个 Transformer 层都用到了残差连接和归一化，原因正在于此。\n\n## 通往大模型\n\n大语言模型就是一个深度神经网络，它的层交替使用**注意力**（在 token 之间交换信息）和**前馈网络**（就是上面讲的神经元层，对每个 token 分别计算），并用反向传播在“预测下一个 token”的损失上训练。接下来读 [Token 与 Embedding](/learn/tokens-embeddings) 和 [Transformer 与注意力机制](/learn/transformer-attention)。"
    }
  }
}