// ⬡ Algorithms Atlas
Deep Learning Foundations
How neural nets are trained: backprop, optimizers, normalization, activations and classic architectures.
21 core2 extended1951–2024
TIMELINE
1951 Stochastic Gra1958 Perceptron1986 Backpropagatio1997 Long Short-Ter1998 Convolutional 2006 Autoencoder2010 Weight Initial2012 Dropout2013 Word2Vec2014 Sequence-to-Se2014 Adam2014 Gated Recurren2014 GloVe2015 Batch Normaliz2015 Residual Netwo2015 U-Net2016 Layer Normaliz2016 GELU2017 AdamW (Decoupl2018 ELMo (Contextu2019 RMSNorm2020 SwiGLU (Gated 2024 Muon Optimizer
01Core algorithms
Learn these first. Sorted by year.
Stochastic Gradient Descent (SGD) Steps downhill using gradients estimated on small mini-batches. 1951core Perceptron The first artificial neuron: weighted sum followed by a threshold. 1958core Backpropagation Computes every parameter's gradient by applying the chain rule backward. 1986core Long Short-Term Memory (LSTM) A gated recurrent network that preserves long-range information. 1997core Convolutional Neural Network (CNN) Slides small learned filters to detect local patterns, composed layer by layer. 1998core Autoencoder Compress the input to a short code and reconstruct it, learning a compact representation. 2006core Weight Initialization (Xavier / He) Scale initial weights by layer width so signals and gradients neither explode nor vanish. 2010core Dropout Randomly drops neurons during training to prevent overfitting. 2012core Word2Vec Learns word vectors by predicting words from context. 2013core Sequence-to-Sequence Learning An encoder reads a sequence and a decoder generates another. 2014core Adam A momentum optimizer with per-parameter adaptive step sizes. 2014core Gated Recurrent Unit (GRU) A leaner gated RNN than LSTM, with two gates for remembering and forgetting. 2014core Batch Normalization Normalizes activations across a batch to stabilize deep training. 2015core Residual Networks (ResNet) Skip connections let signals bypass layers, making very deep nets trainable. 2015core U-Net A U-shaped conv net: downsample, upsample, and skip-connect matching levels. 2015core Layer Normalization Normalizes within each example instead of across the batch. 2016core GELU A smooth activation that gates inputs by their Gaussian probability. 2016core AdamW (Decoupled Weight Decay) Decouples weight decay from Adam's gradient update. 2017core RMSNorm A simplified LayerNorm that only rescales by root-mean-square. 2019core SwiGLU (Gated Linear Units) Adds a gating branch to the feed-forward layer to improve quality. 2020core Muon Optimizer An optimizer that orthogonalizes momentum updates for matrix parameters. 2024core
02Extended
Variants and follow-ups worth knowing once the core is clear.
03Course units that teach these
U12 Calculus & Optimization Derivatives, partial derivatives, the chain rule and gradients — “walking downhill along the gradient”. 12 h2 algorithms1 papers U14 PyTorch & Autograd Tensors, autograd, data loading and the training loop. 10 h2 algorithms0 papers U20 The ML Workflow: Data, Loss & Generalization Train/validation/test splits, overfitting, bias–variance, cross-validation and metrics. 10 h3 algorithms1 paperslesson ready U22 Linear Models, SVM & Nearest Neighbors From the perceptron to maximum margin and the kernel trick, plus k-nearest neighbours. 6 h5 algorithms2 papers U30 Neural Networks & Backpropagation From neurons and multilayer perceptrons to backpropagation and activation functions. 10 h4 algorithms2 paperslesson ready U31 Training Deep Nets SGD/momentum/Adam/AdamW/Muon, learning-rate schedules, BatchNorm/LayerNorm/RMSNorm, dropout and residuals. 10 h10 algorithms4 papers