// Stage 1 · Math & Programming Foundations
Probability, Statistics & Information Theory
Distributions, expectation, Bayes, maximum likelihood, cross-entropy and KL divergence.
about 12 hours3 algorithms0 paperslesson text in progress
01Goals
- Conditional probability and Bayes’ rule
- How maximum likelihood relates to cross-entropy loss
- Entropy, cross-entropy and KL divergence
Do these first: Python, NumPy & Tooling
02Algorithms to master
Each card has the original paper, prerequisites and resources.
03Courses & videos
- mml-book.github.io ↗BooksEN
- statlearning.com ↗BooksEN
- statquest.org ↗VideoEN
04Hands-on: project, code & exercises
Free, official or author-published pages. Most run in the browser or on a free Colab GPU.
PROJECT
Simulate coin flips and estimate the bias by maximum likelihood; then compute cross-entropy and KL between two distributions and verify KL ≥ 0.
- 1Brown University ↗Interactive
Walk through probability, distributions, Bayesian inference and regression with interactive visuals.
- 2
05Self-test
Answer in your head, then open to check.
How does cross-entropy loss relate to maximum likelihood?
Minimising cross-entropy is equivalent to maximising the data’s log-likelihood.
Is KL divergence symmetric?
No: KL(P||Q) ≠ KL(Q||P).
What does raising the temperature do to a softmax distribution?
It flattens it, so sampling becomes more random.
On these routes: Algorithm Researchers · Domain Scientists