{
  "id": "how-llms-learn",
  "module": "training",
  "level": 2,
  "minutes": 25,
  "audiences": [
    "beginner",
    "developer",
    "researcher"
  ],
  "software": [
    "mlx-lm"
  ],
  "text": {
    "en": {
      "title": "How LLMs Learn: Pre-training, Fine-tuning and RLHF Explained",
      "description": "The training pipeline behind ChatGPT-style assistants: pre-training on trillions of tokens, supervised fine-tuning, RLHF with reward models, DPO, constitutional AI, LoRA and reinforcement learning for reasoning.",
      "updated": "2026-10-10",
      "takeaways": [
        "Pre-training teaches a base model language and world knowledge by predicting the next token on trillions of tokens; it uses the vast majority of the compute.",
        "Supervised fine-tuning (SFT) on demonstration conversations turns the base model into an instruction follower.",
        "Preference training — RLHF with a reward model, or simpler methods such as DPO — makes answers more helpful and safer according to human (or AI) judgements.",
        "Reasoning models add reinforcement learning on tasks with checkable answers (maths, code), rewarding correct final results."
      ],
      "faq": [
        {
          "q": "What is the difference between pre-training and fine-tuning?",
          "a": "Pre-training starts from random weights and learns from a huge general corpus. Fine-tuning starts from a pre-trained model and continues training on a much smaller, targeted dataset to change its behaviour or specialise it."
        },
        {
          "q": "What does RLHF stand for and why is it used?",
          "a": "Reinforcement Learning from Human Feedback. People compare model answers, a reward model learns their preferences, and the LLM is optimised to produce answers the reward model scores highly. It improves helpfulness and reduces harmful outputs in ways next-token training alone does not."
        },
        {
          "q": "Is DPO better than RLHF?",
          "a": "DPO (Direct Preference Optimization) reaches the same goal without training a separate reward model or running reinforcement learning, which makes it simpler and cheaper. Which works better depends on data and setup; many open models use DPO or its variants, while frontier labs combine several methods."
        },
        {
          "q": "Can I fine-tune an LLM on my laptop?",
          "a": "Small models, yes, with parameter-efficient methods such as LoRA or QLoRA (on Apple silicon, MLX LM supports LoRA fine-tuning). Often, though, better prompting or RAG solves the problem without any training."
        }
      ],
      "sources": [
        {
          "title": "Ouyang et al. (2022), Training language models to follow instructions with human feedback (InstructGPT)",
          "url": "https://arxiv.org/abs/2203.02155"
        },
        {
          "title": "Christiano et al. (2017), Deep reinforcement learning from human preferences",
          "url": "https://arxiv.org/abs/1706.03741"
        },
        {
          "title": "Hoffmann et al. (2022), Training Compute-Optimal Large Language Models (Chinchilla)",
          "url": "https://arxiv.org/abs/2203.15556"
        },
        {
          "title": "Llama Team, Meta (2024), The Llama 3 Herd of Models",
          "url": "https://arxiv.org/abs/2407.21783"
        },
        {
          "title": "Rafailov et al. (2023), Direct Preference Optimization",
          "url": "https://arxiv.org/abs/2305.18290"
        },
        {
          "title": "Bai et al. (2022), Constitutional AI: Harmlessness from AI Feedback",
          "url": "https://arxiv.org/abs/2212.08073"
        },
        {
          "title": "Hu et al. (2021), LoRA: Low-Rank Adaptation of Large Language Models",
          "url": "https://arxiv.org/abs/2106.09685"
        },
        {
          "title": "DeepSeek-AI (2025), DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning",
          "url": "https://arxiv.org/abs/2501.12948"
        }
      ],
      "url": "https://aimap.page/en/learn/training/how-llms-learn/",
      "markdown": "## The three-stage recipe\n\nModern assistants are built in stages. Each stage starts from the previous model and changes its behaviour with a different kind of data and objective.\n\n```mermaid\nflowchart LR\n  D1[\"Trillions of tokens of web, books, code\"] --> P[\"1. Pre-training → base model\"]\n  P --> S[\"2. Supervised fine-tuning (SFT) → instruction model\"]\n  D2[\"Human-written example dialogues\"] --> S\n  S --> R[\"3. Preference training (RLHF / DPO) → assistant\"]\n  D3[\"Human or AI preference rankings\"] --> R\n  R --> RL[\"Optional: RL on verifiable tasks → reasoning model\"]\n```\n\n## Stage 1: pre-training — learning the world from text\n\nThe model starts with random weights and learns to predict the next token over an enormous corpus ([How Machines Learn](/learn/ml-basics) explains the loop). Meta reports pre-training Llama 3 on about **15 trillion tokens** ([Llama 3 paper](https://arxiv.org/abs/2407.21783)). This stage consumes most of the compute — thousands of GPUs for weeks or months.\n\nHow big should the model be for a given budget? DeepMind's **Chinchilla** study ([Hoffmann et al., 2022](https://arxiv.org/abs/2203.15556)) found that, for compute-optimal training, parameters and training tokens should grow together — roughly 20 tokens per parameter — and that many earlier models were under-trained. Today's models are often trained on far more tokens than that, because a smaller model that has seen more data is cheaper to run.\n\nData quality matters as much as quantity: deduplication, filtering out spam and low-quality text, balancing languages and domains, and adding code and maths all shape what the model can do.\n\nThe result is a **base model**: knowledgeable, but it simply continues text. Asked \"What is the capital of France?\", it may well continue with \"What is the capital of Germany?\".\n\n## Stage 2: supervised fine-tuning — learning the assistant format\n\nNext, the model is trained on tens of thousands to millions of **demonstrations**: a prompt and a high-quality response, written by people or generated and filtered. The loss is the same next-token loss, but only on the response. Now the model learns to answer, follow instructions, use a chat format and refuse clearly harmful requests.\n\n## Stage 3: preference training — RLHF and friends\n\nWriting perfect demonstrations is hard; *judging* which of two answers is better is easier. **Reinforcement learning from human feedback** ([Christiano et al., 2017](https://arxiv.org/abs/1706.03741); applied to LLMs in [InstructGPT](https://arxiv.org/abs/2203.02155)) exploits this:\n\n1. The model produces several answers to the same prompt; labelers **rank** them.\n2. A **reward model** is trained to predict those rankings.\n3. The LLM is optimised with reinforcement learning (PPO in InstructGPT) to produce answers the reward model scores highly, with a penalty for drifting too far from the SFT model.\n\nThe InstructGPT paper reported that people preferred the outputs of a **1.3B-parameter** InstructGPT model over the **175B** GPT-3 base model — alignment training changed usefulness more than a 100× increase in size.\n\nNewer variants simplify or extend this:\n\n- **DPO** – *Direct Preference Optimization* ([Rafailov et al., 2023](https://arxiv.org/abs/2305.18290)) shows the same objective can be optimised directly on preference pairs, with no separate reward model or RL loop.\n- **Constitutional AI / RLAIF** – Anthropic's method ([Bai et al., 2022](https://arxiv.org/abs/2212.08073)) uses a written set of principles and has an AI model critique and rank answers, reducing the need for human labels on harmful content.\n\nPreference training has known side effects: models can learn to please raters rather than be correct (*sycophancy*), or exploit flaws in the reward model (*reward hacking*). See [Alignment Methods](/learn/alignment-methods).\n\n## Reinforcement learning for reasoning\n\nSince 2024, \"reasoning\" models have been trained with reinforcement learning on problems whose answers can be **checked automatically** — maths with known results, code with unit tests. The model writes a long chain of thought, and only the final answer is rewarded. DeepSeek reported that this alone made long, self-checking reasoning emerge in its R1-Zero model, and published the recipe for DeepSeek-R1 ([DeepSeek-AI, 2025](https://arxiv.org/abs/2501.12948)). See [Reasoning Models](/learn/reasoning-models).\n\n## Fine-tuning for your own needs\n\nYou do not have to repeat any of this to adapt a model. **Parameter-efficient fine-tuning** trains a small add-on instead of all weights. **LoRA** ([Hu et al., 2021](https://arxiv.org/abs/2106.09685)) learns small low-rank matrices next to the frozen weights; the paper reports reducing trainable parameters for GPT-3 by about 10,000×. On a Mac, [MLX LM](/apps/mlx-lm) supports LoRA fine-tuning of small open models.\n\nBefore fine-tuning, ask whether you really need it:\n\n| Need | Usually best tool |\n|---|---|\n| Up-to-date or private facts | [RAG](/learn/what-is-rag) |\n| A consistent format or style, a narrow task | Fine-tuning (SFT / LoRA) |\n| Access to live systems and actions | [Tool calling](/learn/tool-calling-mcp) |\n| Better general reasoning | A stronger base or reasoning model |\n\n## What training does not give a model\n\nTraining compresses patterns from data into weights. It does not give the model a reliable record of where a fact came from, knowledge of events after the data cut-off, or a guarantee of truthfulness — which is why grounding, tools and evaluation ([How to Evaluate LLMs](/learn/llm-evaluation)) remain necessary. What the weights actually end up representing is the subject of the next lesson, [What Do LLMs Learn Inside?](/learn/inside-llms)."
    },
    "zh-hans": {
      "title": "大模型如何学习：预训练、微调与 RLHF 一文讲清",
      "description": "ChatGPT 类助手背后的训练流程：在数万亿 token 上预训练、监督微调、基于奖励模型的 RLHF、DPO、宪法式 AI、LoRA，以及面向推理的强化学习。",
      "updated": "2026-10-10",
      "takeaways": [
        "预训练让基座模型通过在数万亿 token 上预测下一个 token 来学会语言和世界知识，消耗了绝大部分算力。",
        "监督微调（SFT）用示范对话继续训练，把基座模型变成会听指令的模型。",
        "偏好训练（带奖励模型的 RLHF，或更简单的 DPO 等方法）按照人类或 AI 的评判，让回答更有用、更安全。",
        "推理模型额外在答案可自动核对的任务（数学、代码）上做强化学习，只奖励正确的最终结果。"
      ],
      "faq": [
        {
          "q": "预训练和微调有什么区别？",
          "a": "预训练从随机参数出发，在海量通用语料上学习；微调从预训练好的模型出发，在小得多、更有针对性的数据上继续训练，用来改变行为或做专门化。"
        },
        {
          "q": "RLHF 是什么，为什么要用？",
          "a": "RLHF 是“基于人类反馈的强化学习”：人对模型的多个回答做比较，奖励模型学习人的偏好，再优化大模型去生成奖励模型打分高的回答。它能提升有用性、减少有害输出，这是单靠预测下一个 token 做不到的。"
        },
        {
          "q": "DPO 比 RLHF 更好吗？",
          "a": "DPO（直接偏好优化）不需要单独训练奖励模型、也不需要跑强化学习，就能达到同样的目标，更简单也更便宜。哪个效果更好取决于数据和设置；很多开源模型用 DPO 或其变体，前沿实验室往往组合使用多种方法。"
        },
        {
          "q": "能在笔记本上微调大模型吗？",
          "a": "小模型可以，用 LoRA、QLoRA 等参数高效的方法（苹果芯片上的 MLX LM 支持 LoRA 微调）。不过很多时候，改进提示词或用 RAG 就能解决问题，根本不用训练。"
        }
      ],
      "sources": [
        {
          "title": "Ouyang 等（2022），Training language models to follow instructions with human feedback（InstructGPT）",
          "url": "https://arxiv.org/abs/2203.02155"
        },
        {
          "title": "Christiano 等（2017），Deep reinforcement learning from human preferences",
          "url": "https://arxiv.org/abs/1706.03741"
        },
        {
          "title": "Hoffmann 等（2022），Training Compute-Optimal Large Language Models（Chinchilla）",
          "url": "https://arxiv.org/abs/2203.15556"
        },
        {
          "title": "Meta Llama 团队（2024），The Llama 3 Herd of Models",
          "url": "https://arxiv.org/abs/2407.21783"
        },
        {
          "title": "Rafailov 等（2023），Direct Preference Optimization（DPO）",
          "url": "https://arxiv.org/abs/2305.18290"
        },
        {
          "title": "Bai 等（2022），Constitutional AI: Harmlessness from AI Feedback",
          "url": "https://arxiv.org/abs/2212.08073"
        },
        {
          "title": "Hu 等（2021），LoRA: Low-Rank Adaptation of Large Language Models",
          "url": "https://arxiv.org/abs/2106.09685"
        },
        {
          "title": "DeepSeek-AI（2025），DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning",
          "url": "https://arxiv.org/abs/2501.12948"
        }
      ],
      "url": "https://aimap.page/zh-hans/learn/training/how-llms-learn/",
      "markdown": "## 三段式训练\n\n现代 AI 助手是分阶段训练出来的。每个阶段都从上一阶段的模型出发，用不同的数据和目标改变它的行为。\n\n```mermaid\nflowchart LR\n  D1[\"数万亿 token 的网页、书籍、代码\"] --> P[\"1. 预训练 → 基座模型\"]\n  P --> S[\"2. 监督微调（SFT）→ 指令模型\"]\n  D2[\"人工编写的示范对话\"] --> S\n  S --> R[\"3. 偏好训练（RLHF / DPO）→ 助手\"]\n  D3[\"人类或 AI 的偏好排序\"] --> R\n  R --> RL[\"可选：在可验证任务上做强化学习 → 推理模型\"]\n```\n\n## 第一阶段：预训练，从文本中学习世界\n\n模型从随机参数出发，在海量语料上学习预测下一个 token（训练循环见 [机器是怎么“学习”的](/learn/ml-basics)）。Meta 公布 Llama 3 的预训练数据约 **15 万亿个 token**（[Llama 3 论文](https://arxiv.org/abs/2407.21783)）。这一阶段消耗了大部分算力：成千上万块 GPU，跑几周到几个月。\n\n给定预算，模型应该多大？DeepMind 的 **Chinchilla** 研究（[Hoffmann 等，2022](https://arxiv.org/abs/2203.15556)）发现，要做到算力最优，参数量和训练 token 数应该同步增长，大约每个参数对应 20 个 token，而此前很多模型“训练不足”。如今的模型常常用远超这个比例的数据训练，因为见过更多数据的小模型，部署起来更便宜。\n\n数据质量和数量同样重要：去重、过滤垃圾和低质量文本、平衡各种语言和领域、加入代码和数学，都会影响模型最终的能力。\n\n结果是一个**基座模型**：知识渊博，但只会续写。你问“法国的首都是哪里？”，它很可能接着写“德国的首都是哪里？”。\n\n## 第二阶段：监督微调，学会助手的样子\n\n接下来，用几万到几百万条**示范数据**继续训练：每条是一个提示和一个高质量回答，由人编写，或由模型生成后筛选。损失仍是“预测下一个 token”，但只在回答部分计算。这样模型学会了回答问题、遵循指令、使用对话格式，并拒绝明显有害的请求。\n\n## 第三阶段：偏好训练，RLHF 及其后继者\n\n写出完美的示范很难，*判断*两个回答哪个更好却容易得多。**基于人类反馈的强化学习（RLHF）**（[Christiano 等，2017](https://arxiv.org/abs/1706.03741)；在 [InstructGPT](https://arxiv.org/abs/2203.02155) 中用于大模型）正是利用了这一点：\n\n1. 模型对同一个提示生成多个回答，标注员给它们**排序**。\n2. 训练一个**奖励模型**来预测这些排序。\n3. 用强化学习（InstructGPT 用的是 PPO）优化大模型，让它生成奖励模型打分高的回答，同时惩罚它偏离 SFT 模型太远。\n\nInstructGPT 论文报告：人们更喜欢 **13 亿参数**的 InstructGPT 的回答，而不是 **1750 亿参数**的 GPT-3 基座模型。对齐训练带来的有用性提升，超过了规模扩大 100 倍。\n\n新的变体对它做了简化或扩展：\n\n- **DPO**，*直接偏好优化*（[Rafailov 等，2023](https://arxiv.org/abs/2305.18290)）：证明同样的目标可以直接在偏好数据对上优化，不需要单独的奖励模型和强化学习循环。\n- **宪法式 AI / RLAIF**：Anthropic 的方法（[Bai 等，2022](https://arxiv.org/abs/2212.08073)）写下一组原则，让 AI 模型自己批评和排序回答，减少了在有害内容上对人工标注的依赖。\n\n偏好训练有已知的副作用：模型可能学会讨好评分者而不是给出正确答案（*谄媚*），或者钻奖励模型的漏洞（*奖励投机*）。见 [对齐方法](/learn/alignment-methods)。\n\n## 面向推理的强化学习\n\n从 2024 年起，“推理模型”会在答案可以**自动核对**的问题上做强化学习：有标准答案的数学题、有单元测试的代码。模型写出很长的思考过程，只有最终答案会被奖励。DeepSeek 报告说，单靠这种方法，R1-Zero 模型就出现了长篇、会自我检查的推理，并公开了 DeepSeek-R1 的训练方法（[DeepSeek-AI，2025](https://arxiv.org/abs/2501.12948)）。见 [推理模型与强化学习](/learn/reasoning-models)。\n\n## 为自己的需求做微调\n\n改造一个模型，并不需要重走上面的流程。**参数高效微调**只训练一个小的附加部分，而不是全部权重。**LoRA**（[Hu 等，2021](https://arxiv.org/abs/2106.09685)）在冻结的权重旁边学习小的低秩矩阵；论文报告在 GPT-3 上把可训练参数减少了约 1 万倍。在 Mac 上，[MLX LM](/apps/mlx-lm) 支持对小型开源模型做 LoRA 微调。\n\n微调之前，先问问自己是不是真的需要：\n\n| 需求 | 通常最合适的工具 |\n|---|---|\n| 最新的或私有的事实 | [RAG](/learn/what-is-rag) |\n| 固定的格式或风格、窄任务 | 微调（SFT / LoRA） |\n| 访问实时系统、执行操作 | [工具调用](/learn/tool-calling-mcp) |\n| 更强的通用推理 | 换更强的基座模型或推理模型 |\n\n## 训练给不了模型什么\n\n训练把数据里的规律压缩进权重。它不会给模型一份可靠的“事实出处记录”，不会让它知道数据截止之后的事，也不保证说真话。所以事实锚定、工具和评测（[怎样评测大模型](/learn/llm-evaluation)）依然必不可少。权重里到底学到了什么，是下一课 [大模型内部学到了什么](/learn/inside-llms) 的主题。"
    }
  }
}