How Machines Learn: Loss Functions and Gradient Descent Explained

Learning = making a number go down

Traditional software is written as rules: if this, then that. Machine learning flips it around: you supply examples and a model with adjustable numbers (parameters), and an algorithm adjusts the numbers until the model's outputs match the examples well.

To make "match well" precise you need a loss function: a formula that measures how wrong the model is on the examples. Training is simply the search for parameter values that make the loss small. Every AI system you have heard of — image classifiers, recommendation engines, ChatGPT — is trained this way.

A tiny worked example

Suppose you want to predict a flat's price from its size. Your model is a straight line:

price = w × size + b

w and b are the two parameters. You have three examples: 50 m² → 100, 80 m² → 160, 100 m² → 200 (in thousands). A common loss for numbers is mean squared error: average of (prediction − truth)².

Start with random guesses, say w = 1, b = 0. Predictions are 50, 80, 100 — far too low — so the loss is large. Now ask: if I increase w a little, does the loss go up or down? That rate of change is the derivative of the loss with respect to w; collected for all parameters it is called the gradient. Here it says "increase w". Take a small step in that direction, recompute, repeat. After enough steps you reach w ≈ 2, b ≈ 0, which fits the data exactly.

flowchart LR
  A["Parameters (random at first)"] --> B["Predict on a batch of examples"]
  B --> C["Loss: how wrong?"]
  C --> D["Gradient: which way reduces loss?"]
  D --> E["Update: step a little downhill"]
  E --> A

The same loop in a few lines of Python (NumPy):

import numpy as np
x = np.array([50, 80, 100.]); y = np.array([100, 160, 200.])
w, b, lr = 1.0, 0.0, 1e-4
for step in range(2000):
    pred = w * x + b
    err = pred - y
    loss = (err ** 2).mean()
    dw = 2 * (err * x).mean()   # d loss / d w
    db = 2 * err.mean()         # d loss / d b
    w, b = w - lr * dw, b - lr * db
print(round(w, 3), round(b, 3), round(loss, 4))

The size of each step, lr, is the learning rate. Too big and the updates overshoot and blow up; too small and learning crawls.

The picture: walking downhill in fog

Imagine the loss as a landscape where every location is a combination of parameter values and height is the loss. You stand somewhere in thick fog and can only feel the slope under your feet. Gradient descent means: feel the slope, take a small step downhill, repeat. With two parameters the landscape is a 3-D surface; with an LLM it has billions of dimensions — but the rule is identical.

Three refinements make this practical at scale:

• Stochastic (mini-batch) gradient descent estimates the slope from a small random batch of examples rather than the full dataset, which is much cheaper (Ruder, 2016).
• Better optimisers such as Adam (Kingma & Ba, 2014) adapt the step size per parameter and add momentum. Adam and its variant AdamW are the default for training LLMs.
• Backpropagation computes the gradient for millions of parameters efficiently using the chain rule — covered in Neural Networks and Backpropagation.

From a straight line to an LLM

For a language model, the setup is the same with three changes:

Cross-entropy is low when the model gave the true next token a high probability and high when it was surprised. Pre-training an LLM is literally running the loop above over trillions of tokens. See How LLMs Learn.

The real goal: generalisation

A model can drive the training loss to zero by memorising examples and still be useless on new inputs — this is overfitting. That is why practitioners split data into training, validation and test sets and judge models on data they never trained on. Large models trained on huge, diverse datasets generalise surprisingly well, but evaluation on fresh, uncontaminated data remains essential (How to Evaluate LLMs).

Try it yourself

Run the Python snippet above (install Python tools quickly with uv: uv run --with numpy python script.py). Then try lr = 1e-3 and watch the loss explode — you have just experienced the most common training failure.