Pre-training, Data, Fine-tuning & Alignment
How LLMs Learn: Pre-training, Fine-tuning and RLHF Explained
Intermediate 25 min For: Beginners / Developers / Researchers
Key takeaways
- ▸Pre-training teaches a base model language and world knowledge by predicting the next token on trillions of tokens; it uses the vast majority of the compute.
- ▸Supervised fine-tuning (SFT) on demonstration conversations turns the base model into an instruction follower.
- ▸Preference training — RLHF with a reward model, or simpler methods such as DPO — makes answers more helpful and safer according to human (or AI) judgements.
- ▸Reasoning models add reinforcement learning on tasks with checkable answers (maths, code), rewarding correct final results.
The three-stage recipe
Modern assistants are built in stages. Each stage starts from the previous model and changes its behaviour with a different kind of data and objective.
flowchart LR D1["Trillions of tokens of web, books, code"] --> P["1. Pre-training → base model"] P --> S["2. Supervised fine-tuning (SFT) → instruction model"] D2["Human-written example dialogues"] --> S S --> R["3. Preference training (RLHF / DPO) → assistant"] D3["Human or AI preference rankings"] --> R R --> RL["Optional: RL on verifiable tasks → reasoning model"]
Stage 1: pre-training — learning the world from text
The model starts with random weights and learns to predict the next token over an enormous corpus (How Machines Learn explains the loop). Meta reports pre-training Llama 3 on about 15 trillion tokens (Llama 3 paper). This stage consumes most of the compute — thousands of GPUs for weeks or months.
How big should the model be for a given budget? DeepMind's Chinchilla study (Hoffmann et al., 2022) found that, for compute-optimal training, parameters and training tokens should grow together — roughly 20 tokens per parameter — and that many earlier models were under-trained. Today's models are often trained on far more tokens than that, because a smaller model that has seen more data is cheaper to run.
Data quality matters as much as quantity: deduplication, filtering out spam and low-quality text, balancing languages and domains, and adding code and maths all shape what the model can do.
The result is a base model: knowledgeable, but it simply continues text. Asked "What is the capital of France?", it may well continue with "What is the capital of Germany?".
Stage 2: supervised fine-tuning — learning the assistant format
Next, the model is trained on tens of thousands to millions of demonstrations: a prompt and a high-quality response, written by people or generated and filtered. The loss is the same next-token loss, but only on the response. Now the model learns to answer, follow instructions, use a chat format and refuse clearly harmful requests.
Stage 3: preference training — RLHF and friends
Writing perfect demonstrations is hard; judging which of two answers is better is easier. Reinforcement learning from human feedback (Christiano et al., 2017; applied to LLMs in InstructGPT) exploits this:
- The model produces several answers to the same prompt; labelers rank them.
- A reward model is trained to predict those rankings.
- The LLM is optimised with reinforcement learning (PPO in InstructGPT) to produce answers the reward model scores highly, with a penalty for drifting too far from the SFT model.
The InstructGPT paper reported that people preferred the outputs of a 1.3B-parameter InstructGPT model over the 175B GPT-3 base model — alignment training changed usefulness more than a 100× increase in size.
Newer variants simplify or extend this:
- DPO – Direct Preference Optimization (Rafailov et al., 2023) shows the same objective can be optimised directly on preference pairs, with no separate reward model or RL loop.
- Constitutional AI / RLAIF – Anthropic's method (Bai et al., 2022) uses a written set of principles and has an AI model critique and rank answers, reducing the need for human labels on harmful content.
Preference training has known side effects: models can learn to please raters rather than be correct (sycophancy), or exploit flaws in the reward model (reward hacking). See Alignment Methods.
Reinforcement learning for reasoning
Since 2024, "reasoning" models have been trained with reinforcement learning on problems whose answers can be checked automatically — maths with known results, code with unit tests. The model writes a long chain of thought, and only the final answer is rewarded. DeepSeek reported that this alone made long, self-checking reasoning emerge in its R1-Zero model, and published the recipe for DeepSeek-R1 (DeepSeek-AI, 2025). See Reasoning Models.
Fine-tuning for your own needs
You do not have to repeat any of this to adapt a model. Parameter-efficient fine-tuning trains a small add-on instead of all weights. LoRA (Hu et al., 2021) learns small low-rank matrices next to the frozen weights; the paper reports reducing trainable parameters for GPT-3 by about 10,000×. On a Mac, MLX LM supports LoRA fine-tuning of small open models.
Before fine-tuning, ask whether you really need it:
| Need | Usually best tool |
|---|---|
| Up-to-date or private facts | RAG |
| A consistent format or style, a narrow task | Fine-tuning (SFT / LoRA) |
| Access to live systems and actions | Tool calling |
| Better general reasoning | A stronger base or reasoning model |
What training does not give a model
Training compresses patterns from data into weights. It does not give the model a reliable record of where a fact came from, knowledge of events after the data cut-off, or a guarantee of truthfulness — which is why grounding, tools and evaluation (How to Evaluate LLMs) remain necessary. What the weights actually end up representing is the subject of the next lesson, What Do LLMs Learn Inside?.
FAQ
What is the difference between pre-training and fine-tuning?
Pre-training starts from random weights and learns from a huge general corpus. Fine-tuning starts from a pre-trained model and continues training on a much smaller, targeted dataset to change its behaviour or specialise it.
What does RLHF stand for and why is it used?
Reinforcement Learning from Human Feedback. People compare model answers, a reward model learns their preferences, and the LLM is optimised to produce answers the reward model scores highly. It improves helpfulness and reduces harmful outputs in ways next-token training alone does not.
Is DPO better than RLHF?
DPO (Direct Preference Optimization) reaches the same goal without training a separate reward model or running reinforcement learning, which makes it simpler and cheaper. Which works better depends on data and setup; many open models use DPO or its variants, while frontier labs combine several methods.
Can I fine-tune an LLM on my laptop?
Small models, yes, with parameter-efficient methods such as LoRA or QLoRA (on Apple silicon, MLX LM supports LoRA fine-tuning). Often, though, better prompting or RAG solves the problem without any training.