// ♞ Reinforcement Learning

Conservative Q-Learning (CQL, Offline RL)

Learns from a fixed dataset by pushing down Q-values of unseen actions.

保守 Q 学习（CQL，离线强化学习） aka Offline RL / 离线强化学习 2020 Advanced extended

IN PLAIN WORDS

Offline RL learns only from logged data, with no new interaction. Ordinary Q-learning over-estimates actions absent from the data, and the policy then chases those imaginary wins. CQL adds a term that pushes their Q-values down, learning a conservative lower bound that keeps the policy within the data’s support.

Used in: Settings limited to logged data – recommendation, healthcare, robotics; D4RL benchmarks

TIMELINE · prerequisites → this → what builds on it

1992 Q-Learning2013 Deep Q-Networ…2020 Conservative Q-Le…

01Original paper

Conservative Q-Learning for Offline Reinforcement Learning

Aviral Kumar, Aurick Zhou, George Tucker, et al. · 2020

arXiv ↗ ✓ Verified · arXiv API

02Where it sits

Learn the left column first; the right column builds on this one.

Learn first

Conservative Q-Learning (CQL, Offline RL)

Then you can learn

Nothing in the atlas builds on it yet

Left: prerequisites · right: builds on this · dashed: related. Full map →

03Learning resources

Official docs, explainers, courses and reference code. All links checked.

04Where the course teaches it

U33 Reinforcement Learning Basics MDPs, Q-learning, DQN, policy gradients, PPO and AlphaZero — the prerequisites for post-training and reasoning models. 14 h12 algorithms4 papers

For AI / data

Verification: verified, checked 2026-10-22.
JSON: /api/learn/algorithms.json (id: cql)

01Original paper

02Where it sits

Learn first

Then you can learn

03Learning resources

Reference code

Courses

Official docs

04Where the course teaches it