// ⬡ Deep Learning Foundations

Stochastic Gradient Descent (SGD)

Steps downhill using gradients estimated on small mini-batches.

随机梯度下降 1951 Beginner core

IN PLAIN WORDS

The loss is a landscape and the gradient points uphill, so step the other way. Mini-batches make it cheap and add helpful noise; momentum turns it into the base of modern optimizers.

Used in: Virtually all neural-network training

TIMELINE · prerequisites → this → what builds on it

1951 Stochastic Gradie…1986 Backpropagati…1992 Q-Learning1992 Policy Gradie…2001 Gradient Boos…2014 Adam

01Original paper

A Stochastic Approximation Method

Robbins et al. · The Annals of Mathematical Statistics · 1951

DOI ↗ ✓ Verified · Crossref API doi:10.1214/aoms/1177729586

02Where it sits

Learn the left column first; the right column builds on this one.

Learn first

Linear Regression

Stochastic Gradient Descent (SGD)

Then you can learn

Left: prerequisites · right: builds on this · dashed: related. Full map →

03Learning resources

Official docs, explainers, courses and reference code. All links checked.

Explainers

Official docs

torch.optim ↗EN

04Where the course teaches it

U12 Calculus & Optimization Derivatives, partial derivatives, the chain rule and gradients — “walking downhill along the gradient”. 12 h2 algorithms1 papers U20 The ML Workflow: Data, Loss & Generalization Train/validation/test splits, overfitting, bias–variance, cross-validation and metrics. 10 h3 algorithms1 paperslesson ready U31 Training Deep Nets SGD/momentum/Adam/AdamW/Muon, learning-rate schedules, BatchNorm/LayerNorm/RMSNorm, dropout and residuals. 10 h10 algorithms4 papers

On learning paths:Zero to Deep Learning

For AI / data

Verification: verified, checked 2026-10-21.
JSON: /api/learn/algorithms.json (id: sgd)