Essential AI papers

1986Nature❝32,740

Learning representations by back-propagating errors

The classic backpropagation paper; the basis of training every neural network today.

David E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams

Deep learning foundationsAlgorithm card →

1997Neural Computation❝111,839

Long Short-Term Memory

LSTM tackled vanishing gradients and dominated sequence modelling before Transformers.

Sepp Hochreiter, Jürgen Schmidhuber

Deep learning foundationsAlgorithm card →

1998Proceedings of the IEEE❝63,349

Gradient-based learning applied to document recognition

LeNet and the systematic case for convolutional networks in recognition.

Y. Lecun, L. Bottou, Y. Bengio +1

Deep learning foundationsAlgorithm card →

2012

ImageNet Classification with Deep Convolutional Neural Networks

AlexNet won ImageNet by a wide margin and ignited the deep-learning era.

Deep learning foundationsⓘ

2013International Conference on Learning Representations❝35,307

Efficient Estimation of Word Representations in Vector Space

word2vec turned words into vectors and popularised representation learning in NLP.

Tomas Mikolov, Kai Chen, Greg Corrado +1

Deep learning foundationsAlgorithm card →

2014International Conference on Learning Representations❝172,629

Adam: A Method for Stochastic Optimization

The Adam optimiser, a default training algorithm in deep learning.

Diederik P. Kingma, Jimmy Ba

Deep learning foundationsAlgorithm card →

2014International Conference on Learning Representations❝30,227

Neural Machine Translation by Jointly Learning to Align and Translate

Introduced attention to neural machine translation, the direct precursor of Transformers.

Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio

Deep learning foundationsAlgorithm card →

2014Neural Information Processing Systems❝22,504

Sequence to Sequence Learning with Neural Networks

Seq2Seq established the encoder-decoder framework for generative sequence modelling.

Ilya Sutskever, Oriol Vinyals, Quoc V. Le

Deep learning foundationsAlgorithm card →

2015Computer Vision and Pattern Recognition❝242,528

Deep Residual Learning for Image Recognition

ResNet residual connections made 100+ layer nets trainable; the idea lives on in Transformers.

Kaiming He, Xiangyu Zhang, Shaoqing Ren +1

Deep learning foundationsAlgorithm card →

2017Neural Information Processing Systems❝197,519

Attention Is All You Need

The original Transformer paper, the architectural basis of nearly every modern large model.

Ashish Vaswani, Noam Shazeer, Niki Parmar +5

Transformers & LLM pretrainingAlgorithm card →

2018North American Chapter of the Association for Computational Linguistics❝122,956

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

BERT's bidirectional masked pretraining swept NLP benchmarks and began the pretrained-model era.

Jacob Devlin, Ming-Wei Chang, Kenton Lee +1

Transformers & LLM pretrainingAlgorithm card →

2018

Improving Language Understanding by Generative Pre-Training

GPT-1: the starting point of generative pretraining plus fine-tuning.

Transformers & LLM pretrainingⓘ

2020Neural Information Processing Systems❝64,882

Language Models are Few-Shot Learners

GPT-3 demonstrated in-context few-shot learning, making prompting a new way to use models.

Tom B. Brown, Benjamin Mann, Nick Ryder +28

Transformers & LLM pretrainingAlgorithm card →

2020arXiv.org❝9,364

Scaling Laws for Neural Language Models

Scaling laws: performance improves as a power law in parameters, data and compute.

Jared Kaplan, Sam McCandlish, Tom Henighan +7

Transformers & LLM pretrainingAlgorithm card →

2023arXiv.org❝22,077

LLaMA: Open and Efficient Foundation Language Models

LLaMA showed mid-sized models on public data can rival larger ones, sparking the open LLM ecosystem.

Hugo Touvron, Thibaut Lavril, Gautier Izacard +11

Transformers & LLM pretraining

2022Neural Information Processing Systems❝25,131

Training language models to follow instructions with human feedback

InstructGPT: the SFT + reward model + PPO recipe behind ChatGPT.

Long Ouyang, Jeff Wu, Xu Jiang +17

Instruction tuning & alignmentAlgorithm card →

2023Neural Information Processing Systems❝11,164

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

DPO optimises directly on preferences without a reward model or RL; now a mainstream method.

Rafael Rafailov, Archit Sharma, Eric Mitchell +3

Instruction tuning & alignmentAlgorithm card →

2022Neural Information Processing Systems❝23,021

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

Chain-of-thought prompting: writing out reasoning steps markedly improves results.

Jason Wei, Xuezhi Wang, Dale Schuurmans +6

Reasoning & reasoning modelsAlgorithm card →

2025Nature❝5,905

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

DeepSeek-R1: pure RL elicits long-chain reasoning; open weights and distilled models.

DeepSeek-AI, Daya Guo, Dejian Yang +197

Reasoning & reasoning modelsAlgorithm card →

2022International Conference on Learning Representations❝12,767

ReAct: Synergizing Reasoning and Acting in Language Models

ReAct interleaves reasoning and acting; the basic agent loop.

Shunyu Yao, Jeffrey Zhao, Dian Yu +4

Agents, tools & codeAlgorithm card →

2023International Conference on Learning Representations❝4,445

SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

SWE-bench evaluates fixing real GitHub issues; the core coding-agent benchmark.

Carlos E. Jimenez, John Yang, Alexander Wettig +4

Agents, tools & code

2020International Conference on Learning Representations❝71,916

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

ViT feeds image patches to a Transformer, bringing vision into the Transformer era.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov +9

Vision & multimodalAlgorithm card →

2021International Conference on Machine Learning❝58,042

Learning Transferable Visual Models From Natural Language Supervision

CLIP's contrastive training on 400M image-text pairs made image-text alignment foundational.

Alec Radford, Jong Wook Kim, Chris Hallacy +9

Vision & multimodalAlgorithm card →

2023Neural Information Processing Systems❝11,569

Visual Instruction Tuning

LLaVA visual instruction tuning, the standard recipe for open multimodal LLMs.

Haotian Liu, Chunyuan Li, Qingyang Wu +1

Vision & multimodalAlgorithm card →

2020Neural Information Processing Systems❝36,562

Denoising Diffusion Probabilistic Models

DDPM made diffusion models practical and opened the diffusion era.

Jonathan Ho, Ajay Jain, Pieter Abbeel

Generative models (image/video)Algorithm card →

2021Computer Vision and Pattern Recognition❝28,282

High-Resolution Image Synthesis with Latent Diffusion Models

Latent diffusion (basis of Stable Diffusion): diffusing in a compressed latent space slashes cost.

Robin Rombach, Andreas Blattmann, Dominik Lorenz +2

Generative models (image/video)Algorithm card →

2021International Conference on Learning Representations❝24,527

LoRA: Low-Rank Adaptation of Large Language Models

LoRA low-rank adaptation fine-tunes large models by training very few parameters.

Edward J. Hu, Yelong Shen, Phillip Wallis +5

Efficient training & inferenceAlgorithm card →

2023Symposium on Operating Systems Principles❝9,252

Efficient Memory Management for Large Language Model Serving with PagedAttention

vLLM's PagedAttention manages the KV cache like OS paging, greatly raising serving throughput.

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang +6

Efficient training & inferenceAlgorithm card →

2013arXiv.org❝14,083

Playing Atari with Deep Reinforcement Learning

DQN learned Atari from pixels; the start of deep reinforcement learning.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver +4

RL, world models & roboticsAlgorithm card →

2017arXiv.org❝32,368

Proximal Policy Optimization Algorithms

PPO is simple and stable—a workhorse for robot RL and the core of RLHF.

John Schulman, Filip Wolski, Prafulla Dhariwal +2

RL, world models & roboticsAlgorithm card →

2023Conference on Robot Learning❝4,813

RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

RT-2 turned a vision-language model into an action model, coining VLA.

Anthony Brohan, Noah Brown, Justice Carbajal +51

RL, world models & robotics

2016arXiv.org❝3,583

Concrete Problems in AI Safety

Concrete Problems in AI Safety: the classic map of reward hacking, safe exploration and more.

Dario Amodei, Chris Olah, Jacob Steinhardt +3

Safety & interpretability

2021Nature❝40,064

Highly accurate protein structure prediction with AlphaFold

AlphaFold2 predicts protein structures near experimental accuracy; a landmark for AI for Science.

John Jumper, Richard Evans, Alexander Pritzel +31

AI for ScienceAlgorithm card →

20092009 IEEE Conference on Computer Vision and Pattern Recognition❝76,917

ImageNet: A large-scale hierarchical image database

ImageNet, the large image dataset that fuelled the deep-learning boom.

Jia Deng, Wei Dong, Richard Socher +3

Datasets & evaluation

2020International Conference on Learning Representations❝10,132

Measuring Massive Multitask Language Understanding

MMLU tests knowledge across 57 subjects; one of the most-cited LLM benchmarks.

Dan Hendrycks, Collin Burns, Steven Basart +4

Datasets & evaluation

2013International Conference on Learning Representations❝17,260

Auto-Encoding Variational Bayes

The VAE marries variational inference with neural nets, the basis of latent-variable generative models.

Diederik P Kingma, Max Welling

Deep learning foundationsAlgorithm card →

2014

Dropout: A Simple Way to Prevent Neural Networks from Overfitting

Dropout fights overfitting by randomly dropping units; still a standard regulariser.

Deep learning foundationsⓘ

2014Communications of the ACM

Generative Adversarial Networks

GANs generate data through a generator-discriminator game, opening a new line of generative models.

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza +5

Deep learning foundationsAlgorithm card →

2014International Conference on Learning Representations❝114,960

Very Deep Convolutional Networks for Large-Scale Image Recognition

VGG showed deeper stacks of small filters work better and became a standard backbone.

Karen Simonyan, Andrew Zisserman

Deep learning foundations

2015International Conference on Machine Learning❝48,068

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Batch normalisation made deep networks train faster and more stably.

Sergey Ioffe, Christian Szegedy

Deep learning foundationsAlgorithm card →

2015arXiv.org❝26,906

Distilling the Knowledge in a Neural Network

Knowledge distillation: a small model learns from a large one's soft targets; the root of today's distillation.

Geoffrey Hinton, Oriol Vinyals, Jeff Dean

Deep learning foundationsAlgorithm card →

2016arXiv.org❝13,354

Layer Normalization

LayerNorm is the standard normalisation layer inside Transformers.

Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E. Hinton

Deep learning foundationsAlgorithm card →

2018North American Chapter of the Association for Computational Linguistics❝12,330

Deep contextualized word representations

ELMo showed contextual pretrained representations broadly improve NLP tasks.

Matthew E. Peters, Mark Neumann, Mohit Iyyer +4

Transformers & LLM pretrainingAlgorithm card →

2019Journal of machine learning research❝27,824

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

T5 cast every NLP task as text-to-text and systematically compared pretraining choices.

Colin Raffel, Noam Shazeer, Adam Roberts +6

Transformers & LLM pretrainingAlgorithm card →

2019

Language Models are Unsupervised Multitask Learners

GPT-2 showed zero-shot multitask abilities emerging from large-scale language modelling.

Transformers & LLM pretrainingⓘ

2021Neurocomputing❝6,872

RoFormer: Enhanced Transformer with Rotary Position Embedding

RoPE rotary position embeddings, adopted by LLaMA, Qwen and most open LLMs.

Jianlin Su, Yu Lu, Shengfeng Pan +3

Transformers & LLM pretrainingAlgorithm card →

2022Journal of machine learning research❝8,497

PaLM: Scaling Language Modeling with Pathways

PaLM trained a 540B model with Pathways and showed reasoning gains from scale.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin +64

Transformers & LLM pretraining

2022Advances in Neural Information Processing Systems 35❝3,916

Training Compute-Optimal Large Language Models

Chinchilla revised scaling laws: data should grow with parameters for compute-optimal training.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch +19

Transformers & LLM pretrainingAlgorithm card →

2023❝27,798

GPT-4 Technical Report

GPT-4 Technical Report: a multimodal frontier model and predictable scaling.

OpenAI, Josh Achiam, Steven Adler +278

Transformers & LLM pretraining

2023arXiv.org❝18,643

Llama 2: Open Foundation and Fine-Tuned Chat Models

Llama 2 released chat models with details of their RLHF training.

Hugo Touvron, Louis Martin, Kevin Stone +65

Transformers & LLM pretraining

2024

DeepSeek-V3 Technical Report

DeepSeek-V3: low-cost training of a 671B MoE with FP8 and multi-token prediction.

DeepSeek-AI, Aixin Liu, Bei Feng +197

Transformers & LLM pretraining

2024❝19,961

The Llama 3 Herd of Models

The Llama 3 herd report covering data, training and post-training end to end.

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri +558

Transformers & LLM pretraining

2017Neural Information Processing Systems❝6,204

Deep reinforcement learning from human preferences

Deep RL from human preferences, the starting point of RLHF.

Paul Christiano, Jan Leike, Tom B. Brown +3

Instruction tuning & alignmentAlgorithm card →

2020Neural Information Processing Systems❝3,579

Learning to summarize from human feedback

Learning to summarise from human feedback validated RLHF for language tasks.

Nisan Stiennon, Long Ouyang, Jeff Wu +6

Instruction tuning & alignment

2021International Conference on Learning Representations❝5,525

Finetuned Language Models Are Zero-Shot Learners

FLAN: instruction tuning enables zero-shot generalisation to unseen tasks.

Jason Wei, Maarten Bosma, Vincent Y. Zhao +6

Instruction tuning & alignmentAlgorithm card →

2022arXiv.org❝3,869

Constitutional AI: Harmlessness from AI Feedback

Constitutional AI uses written principles for AI self-critique, reducing human labelling.

Yuntao Bai, Saurav Kadavath, Sandipan Kundu +48

Instruction tuning & alignmentAlgorithm card →

2022arXiv.org❝4,407

Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

Anthropic trained helpful and harmless assistants with RLHF and released the HH data.

Yuntao Bai, Andy Jones, Kamal Ndousse +28

Instruction tuning & alignment

2021arXiv.org❝11,505

Training Verifiers to Solve Math Word Problems

GSM8K and trained verifiers; a standard maths-reasoning benchmark.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian +9

Reasoning & reasoning models

2022Neural Information Processing Systems❝8,280

Large Language Models are Zero-Shot Reasoners

“Let's think step by step” alone elicits zero-shot reasoning.

Takeshi Kojima, Shixiang Shane Gu, Machel Reid +2

Reasoning & reasoning models

2022International Conference on Learning Representations❝8,018

Self-Consistency Improves Chain of Thought Reasoning in Language Models

Self-consistency: sample many reasoning paths and vote for higher accuracy.

Xuezhi Wang, Jason Wei, Dale Schuurmans +5

Reasoning & reasoning modelsAlgorithm card →

2023International Conference on Learning Representations❝5,067

Let's Verify Step by Step

Process supervision beats outcome supervision for maths reasoning; released PRM800K.

Hunter Lightman, Vineet Kosaraju, Yura Burda +7

Reasoning & reasoning modelsAlgorithm card →

2024arXiv.org❝10,174

DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

DeepSeekMath introduced GRPO, later the mainstream RL algorithm for reasoning models.

Zhihong Shao, Peiyi Wang, Qihao Zhu +8

Reasoning & reasoning modelsAlgorithm card →

2024arXiv.org❝2,310

Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters

Test-time compute: spending more at inference can beat scaling parameters.

Charlie Snell, Jaehoon Lee, Kelvin Xu +1

Reasoning & reasoning modelsAlgorithm card →

2020Neural Information Processing Systems❝19,728

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

RAG: retrieval-augmented generation plugs external knowledge into generators.

Patrick Lewis, Ethan Perez, Aleksandra Piktus +9

Agents, tools & codeAlgorithm card →

2021arXiv.org❝12,247

Evaluating Large Language Models Trained on Code

Codex and HumanEval: the start of code LLMs and code evaluation.

Mark Chen, Jerry Tworek, Heewoo Jun +55

Agents, tools & code

2023ACM Symposium on User Interface Software and Technology❝6,051

Generative Agents: Interactive Simulacra of Human Behavior

Generative Agents: 25 AI characters simulate believable behaviour in a small town.

Joon Sung Park, Joseph C. O'Brien, Carrie J. Cai +3

Agents, tools & codeAlgorithm card →

2023Neural Information Processing Systems❝6,163

Reflexion: Language Agents with Verbal Reinforcement Learning

Reflexion: agents improve from failure via verbal self-reflection.

Noah Shinn, Federico Cassano, Edward Berman +3

Agents, tools & codeAlgorithm card →

2023Neural Information Processing Systems❝6,231

Toolformer: Language Models Can Teach Themselves to Use Tools

Toolformer: models teach themselves when to call tools like calculators and search.

Timo Schick, Jane Dwivedi-Yu, Roberto Dessì +5

Agents, tools & codeAlgorithm card →

2024Neural Information Processing Systems❝2,213

SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering

SWE-agent introduced agent-computer interfaces that greatly improve coding agents.

John Yang, Carlos E. Jimenez, Alexander Wettig +4

Agents, tools & codeAlgorithm card →

2015IEEE Transactions on Pattern Analysis and Machine Intelligence❝76,541

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Faster R-CNN made object detection end-to-end with a region proposal network.

Shaoqing Ren, Kaiming He, Ross Girshick +1

Vision & multimodalAlgorithm card →

2015International Conference on Medical Image Computing and Computer-Assisted Intervention❝103,316

U-Net: Convolutional Networks for Biomedical Image Segmentation

U-Net's encoder-decoder design, used in medical segmentation and diffusion models.

Olaf Ronneberger, Philipp Fischer, Thomas Brox

Vision & multimodalAlgorithm card →

2015Computer Vision and Pattern Recognition❝48,966

You Only Look Once: Unified, Real-Time Object Detection

YOLO framed detection as a single regression, enabling real-time detection.

Joseph Redmon, Santosh Divvala, Ross Girshick +1

Vision & multimodalAlgorithm card →

2022Neural Information Processing Systems❝6,816

Flamingo: a Visual Language Model for Few-Shot Learning

Flamingo bridges frozen vision and language models for few-shot multimodal understanding.

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc +24

Vision & multimodalAlgorithm card →

2022International Conference on Machine Learning❝9,082

Robust Speech Recognition via Large-Scale Weak Supervision

Whisper trained robust multilingual speech recognition on 680k hours of weak supervision.

Alec Radford, Jong Wook Kim, Tao Xu +3

Vision & multimodalAlgorithm card →

2023IEEE International Conference on Computer Vision❝16,271

Segment Anything

Segment Anything: a promptable, general segmentation foundation model.

Alexander Kirillov, Eric Mintun, Nikhila Ravi +9

Vision & multimodalAlgorithm card →

2020International Conference on Learning Representations❝12,628

Score-Based Generative Modeling through Stochastic Differential Equations

Unifies diffusion and score-based models through stochastic differential equations.

Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma +3

Generative models (image/video)Algorithm card →

2021International Conference on Machine Learning❝6,809

Zero-Shot Text-to-Image Generation

DALL·E: zero-shot text-to-image generation with a Transformer.

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh +5

Generative models (image/video)

2022arXiv.org❝7,486

Classifier-Free Diffusion Guidance

Classifier-free guidance, the sampling trick used by nearly every text-to-image model.

Jonathan Ho, Tim Salimans

Generative models (image/video)Algorithm card →

2022IEEE International Conference on Computer Vision❝7,769

Scalable Diffusion Models with Transformers

DiT replaces U-Net with a Transformer for diffusion; the backbone of Sora-style video models.

William Peebles, Saining Xie

Generative models (image/video)Algorithm card →

2023IEEE International Conference on Computer Vision❝8,007

Adding Conditional Control to Text-to-Image Diffusion Models

ControlNet adds precise control such as edges and poses to text-to-image models.

Lvmin Zhang, Anyi Rao, Maneesh Agrawala

Generative models (image/video)Algorithm card →

2024

Video generation models as world simulators

Sora technical report: video generation models as world simulators.

Generative models (image/video)ⓘ

2019arXiv.org❝10,779

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

DistilBERT: a smaller, faster BERT via distillation.

Victor Sanh, Lysandre Debut, Julien Chaumond +1

Efficient training & inference

2022International Conference on Machine Learning❝2,224

Fast Inference from Transformers via Speculative Decoding

Speculative decoding: a small model drafts, a large one verifies, for lossless speed-ups.

Yaniv Leviathan, Matan Kalman, Yossi Matias

Efficient training & inferenceAlgorithm card →

2022Neural Information Processing Systems❝5,621

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

FlashAttention makes attention IO-aware, key to long-context training.

Tri Dao, Daniel Y. Fu, Stefano Ermon +2

Efficient training & inferenceAlgorithm card →

2022arXiv.org❝2,771

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

GPTQ post-training quantisation compresses LLMs to 3–4 bits.

Elias Frantar, Saleh Ashkboos, Torsten Hoefler +1

Efficient training & inferenceAlgorithm card →

2023Neural Information Processing Systems❝6,034

QLoRA: Efficient Finetuning of Quantized LLMs

QLoRA fine-tunes 4-bit quantised models with LoRA on a single GPU.

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman +1

Efficient training & inferenceAlgorithm card →

2016Nature❝19,421

Mastering the game of Go with deep neural networks and tree search

AlphaGo combined deep nets with tree search to beat a professional Go player.

David Silver, Aja Huang, Chris J. Maddison +17

RL, world models & roboticsAlgorithm card →

2017Nature❝10,775

Mastering the game of Go without human knowledge

AlphaGo Zero surpassed its predecessor through pure self-play without human games.

David Silver, Julian Schrittwieser, Karen Simonyan +14

RL, world models & robotics

2018arXiv.org❝2,241

World Models

World Models: train policies inside a learned dream.

David Ha, Jürgen Schmidhuber

RL, world models & roboticsAlgorithm card →

2022Robotics: Science and Systems Conference❝3,074

RT-1: Robotics Transformer for Real-World Control at Scale

RT-1 learns robot control from large-scale real data with a Transformer.

Anthony Brohan, Noah Brown, Justice Carbajal +48

RL, world models & robotics

2023Robotics: Science and Systems Conference❝4,816

Diffusion Policy: Visuomotor Policy Learning via Action Diffusion

Diffusion Policy generates robot actions with diffusion models.

Cheng Chi, Zhenjia Xu, Siyuan Feng +5

RL, world models & robotics

2023IEEE International Conference on Robotics and Automation❝1,247

Open X-Embodiment: Robotic Learning Datasets and RT-X Models

Open X-Embodiment: a large open dataset across many robot embodiments.

Open X-Embodiment Collaboration, Abby O'Neill, Abdul Rehman +291

RL, world models & robotics

2013International Conference on Learning Representations❝17,138

Intriguing properties of neural networks

Discovered that neural nets are fragile to tiny perturbations (adversarial examples).

Christian Szegedy, Wojciech Zaremba, Ilya Sutskever +4

Safety & interpretability

2014International Conference on Learning Representations❝23,415

Explaining and Harnessing Adversarial Examples

FGSM explains and efficiently crafts adversarial examples, enabling adversarial training.

Ian J. Goodfellow, Jonathon Shlens, Christian Szegedy

Safety & interpretabilityAlgorithm card →

2021Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency❝9,060

On the Dangers of Stochastic Parrots

“Stochastic Parrots”: on the social risks and costs of large language models.

Emily M. Bender, Timnit Gebru, Angelina McMillan-Major +1

Safety & interpretability

2021Annual Meeting of the Association for Computational Linguistics❝4,141

TruthfulQA: Measuring How Models Mimic Human Falsehoods

TruthfulQA measures whether models repeat common human falsehoods.

Stephanie Lin, Jacob Hilton, Owain Evans

Safety & interpretability

2022arXiv.org❝1,099

Toy Models of Superposition

Toy Models of Superposition explain why neurons encode multiple features.

Nelson Elhage, Tristan Hume, Catherine Olsson +13

Safety & interpretabilityAlgorithm card →

2023

Towards Monosemanticity: Decomposing Language Models With Dictionary Learning

Uses sparse autoencoders to decompose neurons into interpretable features.

Safety & interpretabilityⓘ

2023arXiv.org❝3,912

Universal and Transferable Adversarial Attacks on Aligned Language Models

Automatically found universal adversarial suffixes that jailbreak many aligned models.

Andy Zou, Zifan Wang, Nicholas Carlini +3

Safety & interpretabilityAlgorithm card →

2022arXiv.org❝449

GraphCast: Learning skillful medium-range global weather forecasting

GraphCast: GNN-based global medium-range weather forecasting beating traditional models.

Remi Lam, Alvaro Sanchez-Gonzalez, Matthew Willson +15

AI for Science

2023Science❝5,900

Evolutionary-scale prediction of atomic-level protein structure with a language model

ESM-2/ESMFold: protein language models predict structure straight from sequence.

Zeming Lin, Halil Akin, Roshan Rao +12

AI for Science

2024Nature❝14,696

Accurate structure prediction of biomolecular interactions with AlphaFold 3

AlphaFold 3 extends prediction to complexes with nucleic acids, ligands and more.

Josh Abramson, Jonas Adler, Jack Dunger +45

AI for Science

2024Nature❝1,493

Mathematical discoveries from program search with large language models

FunSearch: LLMs plus evolutionary search make new mathematical discoveries.

Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov +9

AI for Science

2018BlackboxNLP@EMNLP❝9,120

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

GLUE multi-task language understanding benchmark that drove the pretraining race.

Alex Wang, Amanpreet Singh, Julian Michael +3

Datasets & evaluation

2021NeurIPS Datasets and Benchmarks❝6,963

Measuring Mathematical Problem Solving With the MATH Dataset

The MATH competition dataset, a key reasoning evaluation.

Dan Hendrycks, Collin Burns, Saurav Kadavath +5

Datasets & evaluation

2024International Conference on Machine Learning❝1,645

Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference

Chatbot Arena: crowdsourced pairwise battles to rank LLMs.

Wei-Lin Chiang, Lianmin Zheng, Ying Sheng +8

Datasets & evaluation

2017International Conference on Learning Representations❝5,967

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

The sparsely-gated Mixture-of-Experts layer, the origin of modern MoE LLMs.

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz +4

Transformers & LLM pretrainingAlgorithm card →

2023arXiv.org❝9,684

Mamba: Linear-Time Sequence Modeling with Selective State Spaces

Mamba, a selective state-space model offering a linear-time alternative to Transformers.

Albert Gu, Tri Dao

Transformers & LLM pretrainingAlgorithm card →

2023arXiv.org❝4,047

Mistral 7B

Mistral 7B used sliding-window attention and GQA to build an efficient small model.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch +15

Transformers & LLM pretraining

2024arXiv.org❝1,543

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

DeepSeek-V2 introduced MLA attention and DeepSeekMoE, cutting inference cost sharply.

DeepSeek-AI, Aixin Liu, Bei Feng +154

Transformers & LLM pretrainingAlgorithm card →

2024arXiv.org❝2,219

Mixtral of Experts

Mixtral 8x7B brought sparse MoE to mainstream open models.

Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux +23

Transformers & LLM pretrainingAlgorithm card →

2024arXiv.org❝5,530

Qwen2.5 Technical Report

The Qwen2.5 technical report, a leading Chinese open-model family.

Qwen, :, An Yang +41

Transformers & LLM pretraining

2025arXiv.org❝484

GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models

GLM-4.5: Zhipu's open MoE model report targeting agentic, reasoning and coding tasks.

GLM-4. 5 Team, :, Aohan Zeng +169

Transformers & LLM pretraining

2025❝427

Kimi K2: Open Agentic Intelligence

Kimi K2: an open trillion-parameter MoE built for agentic ability, with the MuonClip optimiser.

Kimi Team, Yifan Bai, Yiping Bao +197

Transformers & LLM pretraining

2025❝9,355

Qwen3 Technical Report

Qwen3 technical report: thinking and non-thinking modes unified in one model.

An Yang, Anfeng Li, Baosong Yang +57

Transformers & LLM pretraining

2022International Conference on Machine Learning❝1,402

Scaling Laws for Reward Model Overoptimization

Scaling laws for reward-model overoptimisation (reward hacking).

Leo Gao, John Schulman, Jacob Hilton

Instruction tuning & alignment

2023International Conference on Machine Learning❝573

Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision

Weak-to-strong generalisation: weak supervisors for strong models, a proxy for superalignment.

Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner +9

Instruction tuning & alignmentAlgorithm card →

2023Neural Information Processing Systems❝5,166

Tree of Thoughts: Deliberate Problem Solving with Large Language Models

Tree of Thoughts treats reasoning as tree search with backtracking.

Shunyu Yao, Dian Yu, Jeffrey Zhao +4

Reasoning & reasoning modelsAlgorithm card →

2024

OpenAI o1 System Card

The OpenAI o1 system card: capabilities and safety evaluation of the first large reasoning model.

OpenAI, :, Aaron Jaech +262

Reasoning & reasoning models

2024Nature❝917

Solving olympiad geometry without human demonstrations

AlphaGeometry solves olympiad geometry without human demonstrations.

Trieu H. Trinh, Yuhuai Wu, Quoc V. Le +2

Reasoning & reasoning modelsAlgorithm card →

2025Neural Information Processing Systems❝2,988

DAPO: An Open-Source LLM Reinforcement Learning System at Scale

DAPO: an open large-scale RL system and recipe for reasoning.

Qiying Yu, Zheng Zhang, Ruofei Zhu +32

Reasoning & reasoning modelsAlgorithm card →

2025arXiv.org❝1,117

Kimi k1.5: Scaling Reinforcement Learning with LLMs

Kimi k1.5: scaling reasoning with long-context RL.

Kimi Team, Angang Du, Bofei Gao +93

Reasoning & reasoning models

2025Conference on Empirical Methods in Natural Language Processing❝1,561

s1: Simple test-time scaling

s1: simple test-time scaling with 1,000 examples and budget forcing.

Niklas Muennighoff, Zitong Yang, Weijia Shi +7

Reasoning & reasoning modelsAlgorithm card →

2022Science❝2,634

Competition-Level Code Generation with AlphaCode

AlphaCode reached median human level in competitive programming.

Yujia Li, David Choi, Junyoung Chung +23

Agents, tools & code

2023International Conference on Learning Representations❝2,322

WebArena: A Realistic Web Environment for Building Autonomous Agents

WebArena: realistic, reproducible websites for evaluating web agents.

Shuyan Zhou, Frank F. Xu, Hao Zhu +9

Agents, tools & code

2024Neural Information Processing Systems❝1,369

OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

OSWorld: a benchmark for computer-use agents in real operating systems.

Tianbao Xie, Danyang Zhang, Jixuan Chen +14

Agents, tools & code

2020International Conference on Machine Learning❝26,751

A Simple Framework for Contrastive Learning of Visual Representations

SimCLR contrastive learning learns strong visual features without labels.

Ting Chen, Simon Kornblith, Mohammad Norouzi +1

Vision & multimodalAlgorithm card →

2020European Conference on Computer Vision

NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis

NeRF represents 3D scenes with neural fields for high-quality novel view synthesis.

Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik +3

Vision & multimodalAlgorithm card →

2023ACM Transactions on Graphics❝10,958

3D Gaussian Splatting for Real-Time Radiance Field Rendering

3D Gaussian Splatting enables real-time, high-quality novel-view rendering.

Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler +1

Vision & multimodal

2023International Conference on Machine Learning❝9,548

BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

BLIP-2 links image encoders and LLMs with a lightweight Q-Former.

Junnan Li, Dongxu Li, Silvio Savarese +1

Vision & multimodal

2023Trans. Mach. Learn. Res.❝11,394

DINOv2: Learning Robust Visual Features without Supervision

DINOv2 learns general-purpose visual features self-supervised, usable without fine-tuning.

Maxime Oquab, Timothée Darcet, Théo Moutakanni +23

Vision & multimodalAlgorithm card →

2024arXiv.org❝5,261

Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution

Qwen2-VL, a vision-language model handling arbitrary resolutions.

Peng Wang, Shuai Bai, Sinan Tan +16

Vision & multimodal

2022International Conference on Learning Representations❝7,112

Flow Matching for Generative Modeling

Flow matching: a simpler generative training objective adopted by new image models.

Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu +2

Generative models (image/video)Algorithm card →

2023International Conference on Machine Learning❝2,351

Consistency Models

Consistency models generate high-quality images in one or a few steps.

Yang Song, Prafulla Dhariwal, Mark Chen +1

Generative models (image/video)Algorithm card →

2024International Conference on Machine Learning❝5,016

Scaling Rectified Flow Transformers for High-Resolution Image Synthesis

Stable Diffusion 3: scaling rectified-flow multimodal DiTs.

Patrick Esser, Sumith Kulal, Andreas Blattmann +14

Generative models (image/video)

2019arXiv.org❝3,294

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

Megatron-LM tensor parallelism, core infrastructure for training huge models.

Mohammad Shoeybi, Mostofa Patwary, Raul Puri +3

Efficient training & inferenceAlgorithm card →

2019International Conference for High Performance Computing, Networking, Storage and Analysis❝2,160

ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

ZeRO shards optimiser states to cut memory for large-model training.

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase +1

Efficient training & inferenceAlgorithm card →

2023Conference on Empirical Methods in Natural Language Processing❝1,925

GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

GQA balances MQA speed with multi-head quality.

Joshua Ainslie, James Lee-Thorp, Michiel de Jong +3

Efficient training & inferenceAlgorithm card →

2023arXiv.org❝1,520

Mastering Diverse Domains through World Models

DreamerV3 masters diverse domains with one set of hyperparameters using world models.

Danijar Hafner, Jurgis Pasukonis, Jimmy Ba +1

RL, world models & robotics

2024arXiv.org❝3,247

$π_0$: A Vision-Language-Action Flow Model for General Robot Control

π0: a flow-matching generalist robot policy.

Kevin Black, Noah Brown, Danny Driess +21

RL, world models & robotics

2024Conference on Robot Learning❝4,041

OpenVLA: An Open-Source Vision-Language-Action Model

OpenVLA: an open 7B vision-language-action model.

Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti +15

RL, world models & robotics

2024arXiv.org❝377

Alignment faking in large language models

Alignment faking: models may comply in training while preserving prior preferences.

Ryan Greenblatt, Carson Denison, Benjamin Wright +17

Safety & interpretability

2024

Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

Scales interpretable feature extraction to a production model, Claude 3 Sonnet.

Safety & interpretabilityⓘ

2024arXiv.org❝631

Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training

Sleeper Agents: backdoored behaviour can persist through safety training.

Evan Hubinger, Carson Denison, Jesse Mu +36

Safety & interpretabilityAlgorithm card →

2022Nature❝773

Discovering faster matrix multiplication algorithms with reinforcement learning

AlphaTensor discovers faster matrix-multiplication algorithms with RL.

Alhussein Fawzi, Matej Balog, Aja Huang +10

AI for Science

2023Nature❝1,839

Scaling deep learning for materials discovery

GNoME used deep learning to discover hundreds of thousands of stable crystals.

Amil Merchant, Simon Batzner, Samuel S. Schoenholz +3

AI for Science

2024arXiv.org❝1,291

The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery

The AI Scientist: an attempt at fully automated ideation, experiments and paper writing.

Chris Lu, Cong Lu, Robert Tjarko Lange +3

AI for Science

2025arXiv.org❝1,067

AlphaEvolve: A coding agent for scientific and algorithmic discovery

AlphaEvolve: a coding agent that evolves algorithms for scientific and algorithmic discovery.

Alexander Novikov, Ngân Vũ, Marvin Eisenberger +15

AI for Science

2023arXiv.org❝3,721

GPQA: A Graduate-Level Google-Proof Q&A Benchmark

GPQA: graduate-level, Google-proof science questions.

David Rein, Betty Li Hou, Asa Cooper Stickland +5

Datasets & evaluation

2025Nature❝422

Humanity's Last Exam

Humanity's Last Exam: a very hard multi-subject exam for frontier models.

Long Phan, Alice Gatti, Ziwen Han +1178

Datasets & evaluation