// ♞ Reinforcement Learning

Actor-Critic & A3C/A2C

An actor picks actions, a critic scores them; the advantage cuts gradient variance.

Actor-Critic 与 A3C/A2C aka A2C / Advantage Actor-Critic 2016 Intermediate core

IN PLAIN WORDS

Plain policy gradients use the whole episode’s return, which is noisy. Actor-critic adds a value network that estimates what a state is worth and updates the policy with “actual minus expected” (the advantage), making learning steadier. A3C runs many environment copies asynchronously; its synchronous variant A2C is more common, and PPO keeps the same frame.

Used in: A standard algorithm (A2C) in Stable-Baselines3 and other libraries; the frame PPO builds on

TIMELINE · prerequisites → this → what builds on it

1992 Policy Gradie…2013 Deep Q-Networ…2016 Actor-Critic & A3…

01Original paper

Asynchronous Methods for Deep Reinforcement Learning

Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, et al. · 2016

arXiv ↗ ✓ Verified · arXiv API

02Where it sits

Learn the left column first; the right column builds on this one.

Learn first

Actor-Critic & A3C/A2C

Then you can learn

Nothing in the atlas builds on it yet

Left: prerequisites · right: builds on this · dashed: related. Full map →

03Learning resources

Official docs, explainers, courses and reference code. All links checked.

04Where the course teaches it

U33 Reinforcement Learning Basics MDPs, Q-learning, DQN, policy gradients, PPO and AlphaZero — the prerequisites for post-training and reasoning models. 14 h12 algorithms4 papers

For AI / data

Verification: verified, checked 2026-10-22.
JSON: /api/learn/algorithms.json (id: a3c)

01Original paper

02Where it sits

Learn first

Then you can learn

03Learning resources

Explainers

Courses

Official docs

04Where the course teaches it