Neural Networks and Backpropagation, Explained Simply

One neuron

An artificial neuron does two things:

1. It multiplies each input by a weight, adds them up and adds a bias: z = w₁x₁ + w₂x₂ + … + b.
2. It passes z through an activation function, for example ReLU: max(0, z).

The weights and bias are the parameters that training adjusts (How Machines Learn). On its own a neuron is a very simple decision rule. The power comes from connecting many.

Layers and depth

Neurons are arranged in layers. Every neuron in a layer reads the outputs of the previous layer. A network with an input layer, one or more hidden layers and an output layer is a multilayer perceptron; "deep" learning simply means many hidden layers.

flowchart LR
  subgraph Input
    x1((x1)); x2((x2)); x3((x3))
  end
  subgraph Hidden1["Hidden layer 1"]
    h1((h)); h2((h)); h3((h)); h4((h))
  end
  subgraph Hidden2["Hidden layer 2"]
    g1((h)); g2((h)); g3((h))
  end
  subgraph Output
    y(("ŷ"))
  end
  x1 --> h1 & h2 & h3 & h4
  x2 --> h1 & h2 & h3 & h4
  x3 --> h1 & h2 & h3 & h4
  h1 & h2 & h3 & h4 --> g1 & g2 & g3
  g1 & g2 & g3 --> y

Why depth helps: each layer can build features out of the previous layer's features. In image networks, early layers detect edges, middle layers textures and parts, later layers whole objects. In language models, early layers handle word pieces and syntax, later layers more abstract meaning (What LLMs Learn Inside).

Why the activation function matters: a stack of purely linear layers is still just one linear function — depth would add nothing. The non-linearity is what lets networks approximate curved, complex relationships. Classic results show that networks with enough hidden units can approximate any reasonable function; in practice, deep networks do so far more efficiently than very wide shallow ones.

Backpropagation: blame assignment

Training needs the gradient of the loss with respect to every weight. Computing each one separately would be hopelessly slow for millions of weights. Backpropagation, popularised by Rumelhart, Hinton and Williams (1986), does it in one backward sweep:

1. Forward pass: compute the output and the loss, remembering intermediate values.
2. Backward pass: starting from the loss, apply the calculus chain rule layer by layer. Each layer receives "how much the loss changes if my output changes" from the layer above and turns it into (a) gradients for its own weights and (b) the same signal for the layer below.
3. Update all weights with gradient descent.

flowchart LR
  I["Input"] -->|"forward"| L1["Layer 1"] -->|"forward"| L2["Layer 2"] -->|"forward"| O["Output → loss"]
  O -.->|"∂loss/∂out"| L2 -.->|"backward"| L1

Intuitively, backprop is blame assignment: it tells each weight how much it contributed to the error. The backward pass costs roughly as much as a couple of forward passes, which is why training networks with billions of weights is feasible at all. Frameworks such as PyTorch and JAX do this automatically ("autograd"); Karpathy's micrograd implements it in about a hundred lines and is the best way to see that there is no magic.

What made deep networks work

Neural networks are old; deep learning took off around 2012, when AlexNet won the ImageNet competition by a wide margin. Three things came together:

• Data: large labelled datasets (ImageNet) and later the whole web.
• Compute: GPUs, which do the required matrix multiplications in parallel (GPUs, Memory and FLOPs).
• Tricks for very deep stacks: ReLU activations, better initialisation, residual connections (He et al., 2015) that let signals skip layers, and normalisation layers. Every Transformer layer in an LLM uses residual connections and normalisation for exactly this reason.

From here to LLMs

A large language model is a deep neural network whose layers alternate between attention (mixing information across tokens) and feed-forward blocks (the neuron layers described above, applied to each token). It is trained by backpropagation on the next-token loss. Continue with Tokens and Embeddings and Transformers and Attention.