// Stage 3 · Deep Learning
Neural Networks & Backpropagation
From neurons and multilayer perceptrons to backpropagation and activation functions.
about 10 hours4 algorithms2 paperslesson text ready
01Goals
- Derive forward and backward passes of a two-layer network by hand
- Understand why activations must be non-linear
- Train an MLP in PyTorch
Do these first: PyTorch & Autograd · The ML Workflow: Data, Loss & Generalization
02Read on this site
03Algorithms to master
Each card has the original paper, prerequisites and resources.
Perceptron The first artificial neuron: weighted sum followed by a threshold. 1958core Backpropagation Computes every parameter's gradient by applying the chain rule backward. 1986core GELU A smooth activation that gates inputs by their Gaussian probability. 2016core Autoencoder Compress the input to a short code and reconstruct it, learning a compact representation. 2006core
04Must-read papers
05Courses & videos
- 3Blue1Brown ↗VideoEN
- Karpathy：Neural Networks: Zero to Hero ↗VideoEN
- cs231n.github.io ↗CoursesEN
- Dive into Deep Learning（D2L） ↗BooksEN
06Hands-on: project, code & exercises
Free, official or author-published pages. Most run in the browser or on a free Colab GPU.
PROJECT
Without a framework, implement a two-layer MLP in NumPy that reaches over 95% accuracy on MNIST.
- 1
- 2
- 3
07Self-test
Answer in your head, then open to check.
A multilayer network without non-linear activations is equivalent to what?
A single linear transformation.
Roughly how expensive is backprop compared with the forward pass?
The same order of magnitude (about 2×).
A common cause of vanishing gradients?
Multiplying many derivatives below 1 in deep stacks, e.g. saturated sigmoids.
On these routes: Algorithm Researchers · Domain Scientists