{
  "id": "pytorch-autograd",
  "module": "deep-learning",
  "level": 2,
  "minutes": 25,
  "audiences": [
    "developer",
    "researcher"
  ],
  "software": [],
  "text": {
    "en": {
      "title": "PyTorch for Beginners: Tensors, Autograd and the Standard Training Loop",
      "description": "The four things you need to train models in PyTorch: tensors and GPUs, how autograd walks the computation graph backwards to get gradients, DataLoader, and the forward → loss → zero_grad → backward → step training loop — with runnable code, common pitfalls and a self-check. Based on the official PyTorch tutorials, the PyTorch paper and the autodiff survey.",
      "updated": "2026-10-23",
      "takeaways": [
        "A tensor is a NumPy-style array that can live on a GPU and supports automatic differentiation; the syntax is nearly identical.",
        "Set requires_grad=True and PyTorch records a computation graph during the forward pass; loss.backward() applies the chain rule back to the parameters and stores gradients in .grad.",
        "PyTorch graphs are dynamic: rebuilt on every forward pass, so models can use ordinary Python if and for.",
        "The standard loop has five steps: forward, compute loss, optimizer.zero_grad(), loss.backward(), optimizer.step(). Gradients accumulate by default, so you zero them every step.",
        "For evaluation, call model.eval() to switch Dropout and BatchNorm behaviour, and wrap the code in torch.no_grad() to skip graph recording — less memory, more speed."
      ],
      "faq": [
        {
          "q": "PyTorch, TensorFlow or JAX?",
          "a": "All work. PyTorch is imperative — your model is a regular Python program, which makes debugging easy — and most code exercises in this site's learning paths use PyTorch, so it is the easiest choice if you follow them."
        },
        {
          "q": "Can I learn PyTorch without a GPU?",
          "a": "Yes. Every example here runs in seconds on a CPU. When you need a GPU, use Google Colab and write device = 'cuda' if torch.cuda.is_available() else 'cpu'."
        },
        {
          "q": "Is autograd the same as backpropagation?",
          "a": "Backpropagation is reverse-mode automatic differentiation applied to neural networks. Autograd is PyTorch's general implementation of reverse-mode autodiff: any program built from tensor operations can be differentiated, not just neural nets."
        },
        {
          "q": "Why isn't my loss going down?",
          "a": "Check three things first: are you calling zero_grad every step; is the learning rate far too high or low; are inputs and labels aligned (print one batch's shapes and values)."
        }
      ],
      "sources": [
        {
          "title": "PyTorch Tutorials: Learn the Basics",
          "url": "https://docs.pytorch.org/tutorials/beginner/basics/intro.html"
        },
        {
          "title": "PyTorch Tutorials: Automatic Differentiation with torch.autograd",
          "url": "https://docs.pytorch.org/tutorials/beginner/basics/autogradqs_tutorial.html"
        },
        {
          "title": "PyTorch Tutorials: Optimizing Model Parameters",
          "url": "https://docs.pytorch.org/tutorials/beginner/basics/optimization_tutorial.html"
        },
        {
          "title": "PyTorch Tutorials: Datasets & DataLoaders",
          "url": "https://docs.pytorch.org/tutorials/beginner/basics/data_tutorial.html"
        },
        {
          "title": "PyTorch docs: Autograd mechanics",
          "url": "https://docs.pytorch.org/docs/stable/notes/autograd.html"
        },
        {
          "title": "PyTorch docs: torch.optim.Adam",
          "url": "https://docs.pytorch.org/docs/stable/generated/torch.optim.Adam.html"
        },
        {
          "title": "PyTorch docs: Reproducibility",
          "url": "https://docs.pytorch.org/docs/stable/notes/randomness.html"
        },
        {
          "title": "PyTorch: Get Started locally",
          "url": "https://pytorch.org/get-started/locally/"
        },
        {
          "title": "Paszke et al. (2019), PyTorch: An Imperative Style, High-Performance Deep Learning Library",
          "url": "https://arxiv.org/abs/1912.01703"
        },
        {
          "title": "Baydin et al. (2015), Automatic differentiation in machine learning: a survey",
          "url": "https://arxiv.org/abs/1502.05767"
        },
        {
          "title": "Kingma & Ba (2014), Adam: A Method for Stochastic Optimization",
          "url": "https://arxiv.org/abs/1412.6980"
        },
        {
          "title": "Andrej Karpathy, micrograd (GitHub)",
          "url": "https://github.com/karpathy/micrograd"
        },
        {
          "title": "Dive into Deep Learning: Automatic Differentiation",
          "url": "https://d2l.ai/chapter_preliminaries/autograd.html"
        }
      ],
      "url": "https://aimap.page/en/learn/deep-learning/pytorch-autograd/",
      "markdown": "## The short answer\n\nPyTorch is one of the mainstream deep-learning frameworks, and most code exercises on this site (nanoGPT, TRL and others) use it. Learning it comes down to two ideas: **tensors record how they were computed, and `loss.backward()` computes the gradient of every parameter for you** — plus one five-step **training loop**. This lesson belongs to the [PyTorch & Autograd](/learn/unit/u14-pytorch) unit; the prerequisite is [Python and NumPy](/learn/python-numpy-setup).\n\n## Step 1: tensors and devices\n\nTensors behave almost exactly like NumPy arrays: they have a shape and a dtype and support broadcasting and matrix maths. The differences: they can run on a GPU, and they support automatic differentiation ([official tutorial](https://docs.pytorch.org/tutorials/beginner/basics/intro.html)).\n\n```python\nimport torch\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\na = torch.randn(3, 4, device=device)\nb = torch.randn(4, 2, device=device)\nc = a @ b            # matrix multiply, shape (3, 2)\n```\n\nTensors in the same operation must be on the same device — one of the most common beginner errors. Install with the selector on the [official Get Started page](https://pytorch.org/get-started/locally/) for your OS and CUDA version rather than a random `pip install`.\n\n## Step 2: how autograd computes gradients\n\nTraining a network means repeatedly \"compute gradients, update parameters\". Deriving every layer by hand is slow and error-prone. **Automatic differentiation** hands it to the program: record each operation during the forward pass, then apply the chain rule step by step in reverse. This reverse mode is the general form of [backpropagation](/learn/algorithms/backprop) in neural networks ([Baydin et al., 2015](https://arxiv.org/abs/1502.05767)).\n\nIn PyTorch you just mark tensors with `requires_grad=True`:\n\n```python\nimport torch\nx = torch.tensor(2.0, requires_grad=True)\nw = torch.tensor(3.0, requires_grad=True)\ny = w * x ** 2 + 1          # y = 3·4 + 1 = 13\ny.backward()\nprint(x.grad, w.grad)       # dy/dx = 2wx = 12, dy/dw = x² = 4\nprint(y.grad_fn)            # <AddBackward0 ...>: y came from an addition\n```\n\nKey points from the docs ([autograd tutorial](https://docs.pytorch.org/tutorials/beginner/basics/autogradqs_tutorial.html), [Autograd mechanics](https://docs.pytorch.org/docs/stable/notes/autograd.html)):\n\n- During the forward pass autograd records a directed acyclic graph (DAG); each result's `grad_fn` points to the operation that produced it.\n- `backward()` starts from the result and applies the chain rule down to the **leaf tensors** (the parameters you created), storing gradients in their `.grad`.\n- **Graphs are dynamic**: after each `backward()` the next forward pass rebuilds the graph from scratch, so models can use Python `if` and `for` freely. The PyTorch paper calls this style imperative — the model is a regular Python program ([Paszke et al., 2019](https://arxiv.org/abs/1912.01703)).\n- **Gradients accumulate** by default; nothing resets them for you:\n\n```python\nw = torch.tensor(1.0, requires_grad=True)\nfor i in range(2):\n    (w * 3).backward()\n    print(w.grad)           # 3 the first time, 6 the second: accumulated\n```\n\nThe best way to see what autograd does under the hood is to build one yourself with Karpathy's [micrograd](https://github.com/karpathy/micrograd): about 100 lines for the autograd engine and 50 for a PyTorch-like neural-net library.\n\n## Step 3: the standard training loop\n\nRewrite last lesson's NumPy linear regression in PyTorch and the hand-written gradient lines disappear:\n\n```python\nimport torch\nfrom torch import nn\n\ntorch.manual_seed(0)\nX = torch.rand(200, 1)\ny = 3 * X + 2 + 0.1 * torch.randn(200, 1)\n\nmodel = nn.Linear(1, 1)\nloss_fn = nn.MSELoss()\nopt = torch.optim.SGD(model.parameters(), lr=0.5)\n\nfor step in range(500):\n    pred = model(X)              # 1. forward\n    loss = loss_fn(pred, y)      # 2. loss\n    opt.zero_grad()              # 3. clear last step's gradients\n    loss.backward()              # 4. backward\n    opt.step()                   # 5. update parameters\n\nprint(model.weight.item(), model.bias.item())   # close to 3 and 2\n```\n\nThese five steps are the optimisation loop from the official tutorial, which notes that gradients add up by default, so you zero them explicitly each iteration to prevent double-counting ([Optimizing Model Parameters](https://docs.pytorch.org/tutorials/beginner/basics/optimization_tutorial.html)). Pre-training and fine-tuning large models are the same five steps — with bigger models, more data and many GPUs.\n\n## Step 4: a fuller version (data loading, Adam, evaluation)\n\nReal projects feed data in mini-batches and usually use [Adam](/learn/algorithms/adam), which adapts each parameter's step size using estimates of the gradient's first and second moments ([Kingma & Ba, 2014](https://arxiv.org/abs/1412.6980)).\n\n```python\nimport torch\nfrom torch import nn\nfrom torch.utils.data import TensorDataset, DataLoader\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nX = torch.randn(1000, 20)\ny = (X[:, 0] + X[:, 1] > 0).long()          # a simple binary task\nloader = DataLoader(TensorDataset(X, y), batch_size=64, shuffle=True)\n\nmodel = nn.Sequential(\n    nn.Linear(20, 64), nn.ReLU(), nn.Dropout(0.1), nn.Linear(64, 2)\n).to(device)\nloss_fn = nn.CrossEntropyLoss()\nopt = torch.optim.Adam(model.parameters(), lr=1e-3)\n\nfor epoch in range(5):\n    model.train()                            # training mode: Dropout active\n    for xb, yb in loader:\n        xb, yb = xb.to(device), yb.to(device)\n        loss = loss_fn(model(xb), yb)\n        opt.zero_grad()\n        loss.backward()\n        opt.step()\n\n    model.eval()                             # eval mode: Dropout off\n    with torch.no_grad():                    # don't record the graph\n        acc = (model(X.to(device)).argmax(1) == y.to(device)).float().mean().item()\n    print(epoch, round(acc, 3))\n```\n\nNotes:\n\n- **`Dataset` and `DataLoader`**: a `Dataset` answers \"what is sample i\"; a `DataLoader` batches, shuffles and loads in parallel ([data tutorial](https://docs.pytorch.org/tutorials/beginner/basics/data_tutorial.html)).\n- **`model.train()` / `model.eval()`**: Dropout and BatchNorm behave differently in training and inference; the official tutorial switches modes in its train and test functions.\n- **`torch.no_grad()`**: evaluation and inference need no gradients; inside this block no graph is recorded, saving memory and time.\n- **Reproducibility**: set `torch.manual_seed`, but fully identical results are not guaranteed across releases, platforms, or CPU vs GPU ([Reproducibility](https://docs.pytorch.org/docs/stable/notes/randomness.html)).\n\n## Common pitfalls\n\n| Symptom | Usual cause | Fix |\n|---|---|---|\n| Loss jumps around or grows | Forgot `zero_grad`, gradients accumulate; or learning rate too high | Check the five-step order; try a 10× smaller learning rate |\n| `Expected all tensors to be on the same device` | Model on GPU, data on CPU (or vice versa) | `.to(device)` the model and every batch |\n| GPU memory keeps growing at inference | No `no_grad`, or storing graph-attached losses in a list | Wrap inference in `torch.no_grad()`; log `loss.item()` |\n| Evaluation results vary run to run | Forgot `model.eval()`, Dropout still on | Switch to eval before evaluating, back to train before training |\n| `.grad` is None | Tensor is not a leaf, or `requires_grad` is unset | Read `.grad` on parameters; use `retain_grad()` for intermediates |\n\n## Try it yourself\n\n1. Train your first image classifier with the official [60-minute blitz](https://docs.pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html).\n2. Solve the 21 [Tensor Puzzles](https://github.com/srush/Tensor-Puzzles), implementing common tensor functions with broadcasting and basic ops only.\n3. Unit project: build a ~100-line autograd engine from scratch following [micrograd](https://github.com/karpathy/micrograd), then write the same two-layer network in PyTorch and check that both produce the same gradients.\n4. For a runnable walkthrough, read the [automatic differentiation](https://d2l.ai/chapter_preliminaries/autograd.html) section of Dive into Deep Learning.\n\nNext, move on to [How Neural Networks Work](/learn/neural-networks) to see what each layer is doing."
    },
    "zh-hans": {
      "title": "PyTorch 入门：张量、自动求导（autograd）与标准训练循环",
      "description": "用 PyTorch 训练模型要懂的四件事：张量与 GPU、autograd 如何沿计算图反向求梯度、DataLoader 取数据，以及“前向 → 损失 → zero_grad → backward → step”的标准训练循环；附可运行代码、常见坑和自测题。依据 PyTorch 官方教程、PyTorch 论文和自动微分综述。",
      "updated": "2026-10-23",
      "takeaways": [
        "张量（tensor）就是能放到 GPU 上、能自动求导的 NumPy 数组，写法几乎一样。",
        "给张量设 requires_grad=True，PyTorch 会在前向计算时记录一张计算图；调用 loss.backward() 就用链式法则把梯度一路算回参数，存进 .grad。",
        "PyTorch 的计算图是动态的：每次前向都重新建图，所以可以直接用 Python 的 if 和 for 写模型。",
        "标准训练循环五步：前向、算损失、optimizer.zero_grad()、loss.backward()、optimizer.step()。梯度默认会累加，所以每步要清零。",
        "评估时用 model.eval() 切换 Dropout、BatchNorm 的行为，再用 torch.no_grad() 关掉梯度记录，省内存也更快。"
      ],
      "faq": [
        {
          "q": "PyTorch 和 TensorFlow/JAX 选哪个？",
          "a": "都能用。PyTorch 的写法是“命令式”的，模型就是普通 Python 程序，调试方便；本站学 AI 路线里的代码练习大多基于 PyTorch，跟着路线学选它最省事。"
        },
        {
          "q": "没有 GPU 能学 PyTorch 吗？",
          "a": "能。本课的例子在 CPU 上几秒就跑完。需要 GPU 时用 Google Colab，代码里只要写 device = 'cuda' if torch.cuda.is_available() else 'cpu'。"
        },
        {
          "q": "autograd 和反向传播是一回事吗？",
          "a": "反向传播是反向模式自动微分在神经网络上的特例。autograd 是 PyTorch 对反向模式自动微分的通用实现，任何由张量运算组成的程序都能求导，不限于神经网络。"
        },
        {
          "q": "为什么我的 loss 不下降？",
          "a": "先查三件事：有没有每步 zero_grad；学习率是否过大或过小；数据和标签是否对齐（打印一个 batch 看看形状和取值）。"
        }
      ],
      "sources": [
        {
          "title": "PyTorch 官方教程：Learn the Basics",
          "url": "https://docs.pytorch.org/tutorials/beginner/basics/intro.html"
        },
        {
          "title": "PyTorch 官方教程：Automatic Differentiation with torch.autograd",
          "url": "https://docs.pytorch.org/tutorials/beginner/basics/autogradqs_tutorial.html"
        },
        {
          "title": "PyTorch 官方教程：Optimizing Model Parameters",
          "url": "https://docs.pytorch.org/tutorials/beginner/basics/optimization_tutorial.html"
        },
        {
          "title": "PyTorch 官方教程：Datasets & DataLoaders",
          "url": "https://docs.pytorch.org/tutorials/beginner/basics/data_tutorial.html"
        },
        {
          "title": "PyTorch 文档：Autograd mechanics",
          "url": "https://docs.pytorch.org/docs/stable/notes/autograd.html"
        },
        {
          "title": "PyTorch 文档：torch.optim.Adam",
          "url": "https://docs.pytorch.org/docs/stable/generated/torch.optim.Adam.html"
        },
        {
          "title": "PyTorch 文档：Reproducibility（随机性）",
          "url": "https://docs.pytorch.org/docs/stable/notes/randomness.html"
        },
        {
          "title": "PyTorch 官方：本地安装（Get Started）",
          "url": "https://pytorch.org/get-started/locally/"
        },
        {
          "title": "Paszke 等（2019），PyTorch: An Imperative Style, High-Performance Deep Learning Library",
          "url": "https://arxiv.org/abs/1912.01703"
        },
        {
          "title": "Baydin 等（2015），Automatic differentiation in machine learning: a survey",
          "url": "https://arxiv.org/abs/1502.05767"
        },
        {
          "title": "Kingma & Ba（2014），Adam: A Method for Stochastic Optimization",
          "url": "https://arxiv.org/abs/1412.6980"
        },
        {
          "title": "Andrej Karpathy，micrograd（GitHub）",
          "url": "https://github.com/karpathy/micrograd"
        },
        {
          "title": "《动手学深度学习》：自动微分",
          "url": "https://d2l.ai/chapter_preliminaries/autograd.html"
        }
      ],
      "url": "https://aimap.page/zh-hans/learn/deep-learning/pytorch-autograd/",
      "markdown": "## 一句话回答\n\nPyTorch 是主流的深度学习框架之一，本站大多数代码练习（nanoGPT、TRL、micrograd 的对照实现等）都用它。学会它只需要抓住两点：**张量会自动记录计算过程，`loss.backward()` 能自动算出所有参数的梯度**；以及一个五步的**标准训练循环**。本课属于 [PyTorch 入门与自动求导](/learn/unit/u14-pytorch) 单元，前置是 [Python 与 NumPy](/learn/python-numpy-setup)。\n\n## 第一步：张量与设备\n\n张量（tensor）的用法和 NumPy 数组几乎一样：有形状、数据类型，支持广播和矩阵运算。区别在两点：它可以放到 GPU 上计算，并且能自动求导（[官方教程](https://docs.pytorch.org/tutorials/beginner/basics/intro.html)）。\n\n```python\nimport torch\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\na = torch.randn(3, 4, device=device)\nb = torch.randn(4, 2, device=device)\nc = a @ b            # 矩阵乘法，形状 (3, 2)\n```\n\n参与同一次运算的张量必须在同一个设备上，否则会报错。这是新手最常见的错误之一。安装方法按 [官方安装页](https://pytorch.org/get-started/locally/) 选择系统和 CUDA 版本，不要随手 `pip install` 一个不匹配的版本。\n\n## 第二步：autograd 是怎么算梯度的\n\n训练神经网络就是反复“算梯度、改参数”。手推每一层的导数既慢又容易错，**自动微分**（automatic differentiation）把它交给程序：前向计算时记下每一步运算，再沿相反方向用链式法则逐步求导。这种“反向模式”自动微分正是神经网络里 [反向传播](/learn/algorithms/backprop) 的一般形式（[Baydin 等，2015](https://arxiv.org/abs/1502.05767)）。\n\n在 PyTorch 里只要把需要求导的张量标成 `requires_grad=True`：\n\n```python\nimport torch\nx = torch.tensor(2.0, requires_grad=True)\nw = torch.tensor(3.0, requires_grad=True)\ny = w * x ** 2 + 1          # y = 3·4 + 1 = 13\ny.backward()\nprint(x.grad, w.grad)       # dy/dx = 2wx = 12，dy/dw = x² = 4\nprint(y.grad_fn)            # <AddBackward0 ...>：y 是由加法得到的\n```\n\n官方教程讲了几个要点（[autograd 教程](https://docs.pytorch.org/tutorials/beginner/basics/autogradqs_tutorial.html)、[Autograd mechanics](https://docs.pytorch.org/docs/stable/notes/autograd.html)）：\n\n- 前向时，autograd 把运算记成一张有向无环图（DAG），每个结果张量的 `grad_fn` 指向生成它的运算。\n- `backward()` 从结果出发，沿图用链式法则算到**叶子张量**（你自己创建的参数），梯度存进叶子的 `.grad`。\n- **图是动态的**：每次 `backward()` 之后，下一次前向会从头重建计算图。所以模型里可以直接写 Python 的 `if`、`for`，每一步走的分支不同也没关系。PyTorch 论文把这种风格称为“命令式”：模型就是一段普通的 Python 程序（[Paszke 等，2019](https://arxiv.org/abs/1912.01703)）。\n- **梯度默认会累加**，不会自动清零：\n\n```python\nw = torch.tensor(1.0, requires_grad=True)\nfor i in range(2):\n    (w * 3).backward()\n    print(w.grad)           # 第一次 3，第二次 6：累加了\n```\n\n想看清 autograd 在底层做了什么，最好的办法是跟着 Karpathy 的 [micrograd](https://github.com/karpathy/micrograd) 自己写一个：它的自动求导引擎约 100 行，神经网络库约 50 行，接口模仿 PyTorch。\n\n## 第三步：标准训练循环\n\n把上一课用 NumPy 手写梯度的线性回归改成 PyTorch，梯度那两行就消失了：\n\n```python\nimport torch\nfrom torch import nn\n\ntorch.manual_seed(0)\nX = torch.rand(200, 1)\ny = 3 * X + 2 + 0.1 * torch.randn(200, 1)\n\nmodel = nn.Linear(1, 1)\nloss_fn = nn.MSELoss()\nopt = torch.optim.SGD(model.parameters(), lr=0.5)\n\nfor step in range(500):\n    pred = model(X)              # 1. 前向\n    loss = loss_fn(pred, y)      # 2. 算损失\n    opt.zero_grad()              # 3. 清空上一步的梯度\n    loss.backward()              # 4. 反向求梯度\n    opt.step()                   # 5. 按梯度更新参数\n\nprint(model.weight.item(), model.bias.item())   # 接近 3 和 2\n```\n\n这五步就是 PyTorch 官方教程里的优化循环。教程特别说明：梯度默认会累加，为了避免重复计算，每次迭代都要显式清零（[Optimizing Model Parameters](https://docs.pytorch.org/tutorials/beginner/basics/optimization_tutorial.html)）。大模型的预训练和微调，本质上也是这五步，只是模型更大、数据更多、跑在多张 GPU 上。\n\n## 第四步：完整一点的写法（数据加载、Adam 与评估）\n\n真实项目里，数据要分批（mini-batch）送进模型，优化器多用 [Adam](/learn/algorithms/adam)。Adam 根据梯度的一阶矩和二阶矩估计自适应地调整每个参数的步长（[Kingma & Ba，2014](https://arxiv.org/abs/1412.6980)）。\n\n```python\nimport torch\nfrom torch import nn\nfrom torch.utils.data import TensorDataset, DataLoader\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nX = torch.randn(1000, 20)\ny = (X[:, 0] + X[:, 1] > 0).long()          # 一个简单的二分类任务\nloader = DataLoader(TensorDataset(X, y), batch_size=64, shuffle=True)\n\nmodel = nn.Sequential(\n    nn.Linear(20, 64), nn.ReLU(), nn.Dropout(0.1), nn.Linear(64, 2)\n).to(device)\nloss_fn = nn.CrossEntropyLoss()\nopt = torch.optim.Adam(model.parameters(), lr=1e-3)\n\nfor epoch in range(5):\n    model.train()                            # 训练模式：Dropout 生效\n    for xb, yb in loader:\n        xb, yb = xb.to(device), yb.to(device)\n        loss = loss_fn(model(xb), yb)\n        opt.zero_grad()\n        loss.backward()\n        opt.step()\n\n    model.eval()                             # 评估模式：Dropout 关闭\n    with torch.no_grad():                    # 不记录计算图\n        acc = (model(X.to(device)).argmax(1) == y.to(device)).float().mean().item()\n    print(epoch, round(acc, 3))\n```\n\n几个要点：\n\n- **`Dataset` 和 `DataLoader`**：`Dataset` 负责“第 i 个样本是什么”，`DataLoader` 负责分批、打乱和并行读取（[数据加载教程](https://docs.pytorch.org/tutorials/beginner/basics/data_tutorial.html)）。\n- **`model.train()` / `model.eval()`**：Dropout 和 BatchNorm 在训练和推理时行为不同，官方教程在训练和测试函数里分别切换模式。\n- **`torch.no_grad()`**：评估和推理时不需要梯度，包在这个块里可以不记录计算图，省内存、也更快。\n- **可复现**：设 `torch.manual_seed`，但在不同硬件和版本上，完全一致的结果并不保证（[Reproducibility](https://docs.pytorch.org/docs/stable/notes/randomness.html)）。\n\n## 常见坑\n\n| 现象 | 常见原因 | 怎么办 |\n|---|---|---|\n| loss 忽高忽低或越训越大 | 忘了 `zero_grad`，梯度在累加；或学习率太大 | 检查循环五步顺序；把学习率调小 10 倍试试 |\n| `Expected all tensors to be on the same device` | 模型在 GPU、数据在 CPU（或相反） | 模型 `.to(device)`，每个 batch 也 `.to(device)` |\n| 推理时显存不断上涨 | 没用 `no_grad`，或把带图的 loss 存进列表 | 推理包进 `torch.no_grad()`；记录数值用 `loss.item()` |\n| 评估结果每次都不一样 | 忘了 `model.eval()`，Dropout 还开着 | 评估前切到 eval，训练前切回 train |\n| `.grad` 是 None | 张量不是叶子，或没设 `requires_grad` | 只看参数的 `.grad`；中间结果需要时用 `retain_grad()` |\n\n## 动手试试\n\n1. 跟着官方 [60 分钟入门](https://docs.pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html) 训练第一个图像分类器。\n2. 做 [Tensor Puzzles](https://github.com/srush/Tensor-Puzzles) 的 21 道题，只用广播和基本运算实现常见张量函数。\n3. 本单元项目：跟着 [micrograd](https://github.com/karpathy/micrograd) 从零写一个约 100 行的自动求导引擎，再用 PyTorch 写同一个两层网络，对比两边算出的梯度是否一致。\n4. 想要中文、可直接运行的讲解，读《动手学深度学习》的 [自动微分](https://d2l.ai/chapter_preliminaries/autograd.html) 一节（中文版见 zh.d2l.ai）。\n\n学完这一课，就可以进入 [神经网络是怎么工作的](/learn/neural-networks)，看清每一层在做什么。"
    }
  }
}