// Algorithm learning path

Zero to Deep Learning

From linear regression all the way to ResNet and the precursors of the Transformer — build intuition for how models are trained.

Level

Start at step 1 ↓ See it on the map →

4 steps24 algorithms5 course units

01Steps

Each step builds on the one before. Open any card for the paper and resources.

Start: learning is minimising a loss
3 algorithms

Linear Regression Fits a line (hyperplane) between inputs and a numeric output. core Logistic Regression Turns a linear score into a probability with a sigmoid for binary classification. 1958core Stochastic Gradient Descent (SGD) Steps downhill using gradients estimated on small mini-batches. 1951core
The classical model toolbox
8 algorithms

Decision Trees (CART) Splits data with a sequence of yes/no questions into pure groups. core Random Forest Trains many randomized decision trees and lets them vote. 2001core Gradient Boosting (GBDT) Adds small trees one by one, each fixing the previous ensemble's errors. 2001core XGBoost A highly engineered, scalable gradient-boosted tree system. 2016core Support Vector Machine Finds the decision boundary with the widest margin between classes. 1995core k-Nearest Neighbors Predicts by majority vote of the k closest training examples. 1967core k-Means Clustering Alternates assigning points and recomputing centers to form k clusters. 1982core Principal Component Analysis (PCA) Reduces dimensions by keeping directions of greatest variance. 1901core
Neural networks and training
7 algorithms

Perceptron The first artificial neuron: weighted sum followed by a threshold. 1958core Backpropagation Computes every parameter's gradient by applying the chain rule backward. 1986core Adam A momentum optimizer with per-parameter adaptive step sizes. 2014core AdamW (Decoupled Weight Decay) Decouples weight decay from Adam's gradient update. 2017core Dropout Randomly drops neurons during training to prevent overfitting. 2012core Batch Normalization Normalizes activations across a batch to stabilize deep training. 2015core Layer Normalization Normalizes within each example instead of across the batch. 2016core
Classic network architectures
6 algorithms

Convolutional Neural Network (CNN) Slides small learned filters to detect local patterns, composed layer by layer. 1998core Residual Networks (ResNet) Skip connections let signals bypass layers, making very deep nets trainable. 2015core Long Short-Term Memory (LSTM) A gated recurrent network that preserves long-range information. 1997core Sequence-to-Sequence Learning An encoder reads a sequence and a decoder generates another. 2014core Word2Vec Learns word vectors by predicting words from context. 2013core Attention (Bahdanau) Lets the decoder look back at the most relevant input positions. 2014core

02Course units for this path

Goals, projects and self-tests that go with these algorithms.

U20 The ML Workflow: Data, Loss & Generalization Train/validation/test splits, overfitting, bias–variance, cross-validation and metrics. 10 h3 algorithms1 paperslesson ready U21 Trees & Ensembles Decision trees, random forests, gradient boosting, XGBoost/LightGBM — the kings of tabular data. 8 h6 algorithms3 papers U30 Neural Networks & Backpropagation From neurons and multilayer perceptrons to backpropagation and activation functions. 10 h4 algorithms2 paperslesson ready U31 Training Deep Nets SGD/momentum/Adam/AdamW/Muon, learning-rate schedules, BatchNorm/LayerNorm/RMSNorm, dropout and residuals. 10 h10 algorithms4 papers U32 CNNs, RNNs and the Birth of Attention Convolutional networks, LSTMs, Seq2Seq, word vectors and Bahdanau attention. 10 h7 algorithms5 papers

01Steps

Start: learning is minimising a loss

The classical model toolbox

Neural networks and training

Classic network architectures

02Course units for this path

03Other paths