Skip to content
English

Switch theme

Discover▾

Not sure where to start? This week’s hottest software, models and stacks in three minutes.

AI Apps▾

AI software people use: from assistants like ChatGPT to tools for coding, video and workflows, picked by purpose.

For Agents▾

Software for AI: tools that let your agent (Claude Code, Codex, Cursor…) browse, read docs, run code and remember.

Models▾

Choose models, estimate cost, self-host: prices, rankings, benchmarks, free tiers and compute.

Solutions▾

A goal or an error message? Ready-made stacks, steps and fixes by scenario.

Learn▾

A systematic course from zero to the frontier, with an algorithm library, must-read papers and tutorials.

Frontier▾

Daily AI news, weekly highlights, the latest papers and how AI is changing every discipline.

// Stage 3 · Deep Learning

Training Deep Nets

SGD/momentum/Adam/AdamW/Muon, learning-rate schedules, BatchNorm/LayerNorm/RMSNorm, dropout and residuals.

What you will learn ↓ Code & exercises ↓

about 10 hours10 algorithms4 paperslesson text in progress

01Goals

Do these first: Neural Networks & Backpropagation

02Algorithms to master

Each card has the original paper, prerequisites and resources.

Stochastic Gradient Descent (SGD) Steps downhill using gradients estimated on small mini-batches. 1951core Adam A momentum optimizer with per-parameter adaptive step sizes. 2014core AdamW (Decoupled Weight Decay) Decouples weight decay from Adam's gradient update. 2017core Muon Optimizer An optimizer that orthogonalizes momentum updates for matrix parameters. 2024core Batch Normalization Normalizes activations across a batch to stabilize deep training. 2015core Layer Normalization Normalizes within each example instead of across the batch. 2016core RMSNorm A simplified LayerNorm that only rescales by root-mean-square. 2019core Dropout Randomly drops neurons during training to prevent overfitting. 2012core Residual Networks (ResNet) Skip connections let signals bypass layers, making very deep nets trainable. 2015core Weight Initialization (Xavier / He) Scale initial weights by layer width so signals and gradients neither explode nor vanish. 2010core

03Must-read papers

04Courses & videos

05Hands-on: project, code & exercises

Free, official or author-published pages. Most run in the browser or on a free Colab GPU.

PROJECT

Run ablations on CIFAR-10: remove BN, remove residuals, swap the optimiser; train each for 10 epochs and record the curves.

  1. 1
    github.com ↗Notebooks

    Do makemore parts 3–4: initialisation, BatchNorm and manual backprop.

  2. 2
    Kaggle Learn：Intro to Deep Learning ↗Exercises

    Practise SGD, overfitting, dropout and batch norm.

  3. 3
    github.com ↗Notebooks

    Run the Optimisation chapter notebooks comparing SGD, momentum and Adam.

06Self-test

Answer in your head, then open to check.

How does AdamW differ from Adam + L2?

AdamW decays weights directly instead of through the adaptive scaling.

What is warmup for?

Using a small learning rate early in training to avoid instability.

What does RMSNorm drop compared with LayerNorm?

Mean subtraction and the bias.

On these routes: Algorithm Researchers

Data updated: AI algorithms & curriculum(UTC+8)