Skip to content
English

Switch theme

Discover▾

Not sure where to start? This week’s hottest software, models and stacks in three minutes.

AI Apps▾

AI software people use: from assistants like ChatGPT to tools for coding, video and workflows, picked by purpose.

For Agents▾

Software for AI: tools that let your agent (Claude Code, Codex, Cursor…) browse, read docs, run code and remember.

Models▾

Choose models, estimate cost, self-host: prices, rankings, benchmarks, free tiers and compute.

Solutions▾

A goal or an error message? Ready-made stacks, steps and fixes by scenario.

Learn▾

A systematic course from zero to the frontier, with an algorithm library, must-read papers and tutorials.

Frontier▾

Daily AI news, weekly highlights, the latest papers and how AI is changing every discipline.

// Stage 3 · Deep Learning

Reinforcement Learning Basics

MDPs, Q-learning, DQN, policy gradients, PPO and AlphaZero — the prerequisites for post-training and reasoning models.

What you will learn ↓ Code & exercises ↓

about 14 hours12 algorithms4 paperslesson text in progress

01Goals

Do these first: Training Deep Nets · Probability, Statistics & Information Theory

02Algorithms to master

Each card has the original paper, prerequisites and resources.

Q-Learning Learns the long-term value of each action in each state. 1992core Deep Q-Network (DQN) Approximates Q with a CNN to play Atari from pixels. 2013core Policy Gradient (REINFORCE) Directly increases the probability of actions that led to high return. 1992core Proximal Policy Optimization (PPO) A stable, simple policy-gradient method that clips update size. 2017core Soft Actor-Critic (SAC) Off-policy actor-critic maximizing both return and entropy. 2018ext AlphaGo / AlphaZero (Self-Play + MCTS) Neural evaluation plus tree search, learned from scratch by self-play. 2016core MuZero Plans without knowing the rules by learning a value-focused model. 2019ext Actor-Critic & A3C/A2C An actor picks actions, a critic scores them; the advantage cuts gradient variance. 2016core Trust Region Policy Optimization (TRPO) Caps each policy update by a KL “trust region” for monotonic improvement. 2015ext DDPG / TD3 (Continuous Control) Brings DQN-style learning to continuous actions with a deterministic actor and a critic. 2015core Conservative Q-Learning (CQL, Offline RL) Learns from a fixed dataset by pushing down Q-values of unseen actions. 2020ext Decision Transformer RL as sequence modelling: condition on a target return and predict the next action. 2021ext

03Must-read papers

04Courses & videos

05Hands-on: project, code & exercises

Free, official or author-published pages. Most run in the browser or on a free Colab GPU.

PROJECT

Train CartPole in Gymnasium: first a tabular/DQN version, then PPO; compare sample efficiency.

  1. 1
    huggingface.co ↗Courses

    Do the hands-on units of the Hugging Face Deep RL course (Q-learning, DQN, PPO).

  2. 2
    gymnasium.farama.org ↗Official docs

    Run your own agent in CartPole, LunarLander and other standard environments.

  3. 3
    github.com ↗Reference code

    Read the single-file DQN/PPO/TD3 code and tweak hyperparameters to see learning curves.

06Self-test

Answer in your head, then open to check.

On-policy vs off-policy?

Whether updates only use data produced by the current policy.

Why does PPO clip the probability ratio?

To limit the size of each update and keep training stable.

Where does AlphaZero’s training data come from?

Self-play.

On these routes: Algorithm Researchers

Data updated: AI algorithms & curriculum(UTC+8)