// ♞ Algorithms Atlas
Reinforcement Learning
From Q-learning to PPO, AlphaZero and MuZero — learning decisions from trial and reward.
7 core5 extended1992–2021
TIMELINE
01Core algorithms
Learn these first. Sorted by year.
Q-Learning Learns the long-term value of each action in each state. 1992core Policy Gradient (REINFORCE) Directly increases the probability of actions that led to high return. 1992core Deep Q-Network (DQN) Approximates Q with a CNN to play Atari from pixels. 2013core DDPG / TD3 (Continuous Control) Brings DQN-style learning to continuous actions with a deterministic actor and a critic. 2015core AlphaGo / AlphaZero (Self-Play + MCTS) Neural evaluation plus tree search, learned from scratch by self-play. 2016core Actor-Critic & A3C/A2C An actor picks actions, a critic scores them; the advantage cuts gradient variance. 2016core Proximal Policy Optimization (PPO) A stable, simple policy-gradient method that clips update size. 2017core
02Extended
Variants and follow-ups worth knowing once the core is clear.
Trust Region Policy Optimization (TRPO) Caps each policy update by a KL “trust region” for monotonic improvement. 2015ext Soft Actor-Critic (SAC) Off-policy actor-critic maximizing both return and entropy. 2018ext MuZero Plans without knowing the rules by learning a value-focused model. 2019ext Conservative Q-Learning (CQL, Offline RL) Learns from a fixed dataset by pushing down Q-values of unseen actions. 2020ext Decision Transformer RL as sequence modelling: condition on a target return and predict the next action. 2021ext