// Learn AI
AI Algorithms Atlas
223 algorithms · 20 areas · original paper for each. Look up what an algorithm is, what to learn first and where it is used — from 1901 to today.
220 papers verified via arXiv API / Crossref / official pages3 still being checked (marked)updated 2026-10-22
01Find an algorithm
Search Chinese or English names and aliases; filter by area, difficulty and era.
- Linear Regression · Classical Machine Learning
Fits a line (hyperplane) between inputs and a numeric output. — - Logistic Regression · Classical Machine Learning
Turns a linear score into a probability with a sigmoid for binary classification. 1958 - k-Nearest Neighbors · Classical Machine Learning
Predicts by majority vote of the k closest training examples. 1967 - Decision Trees (CART) · Classical Machine Learning
Splits data with a sequence of yes/no questions into pure groups. — - Random Forest · Classical Machine Learning
Trains many randomized decision trees and lets them vote. 2001 - Gradient Boosting (GBDT) · Classical Machine Learning
Adds small trees one by one, each fixing the previous ensemble's errors. 2001 - XGBoost · Classical Machine Learning
A highly engineered, scalable gradient-boosted tree system. 2016 - LightGBM · Classical Machine Learning
Gradient boosting accelerated by histograms and leaf-wise growth. 2017 - Support Vector Machine · Classical Machine Learning
Finds the decision boundary with the widest margin between classes. 1995 - k-Means Clustering · Classical Machine Learning
Alternates assigning points and recomputing centers to form k clusters. 1982 - Principal Component Analysis (PCA) · Classical Machine Learning
Reduces dimensions by keeping directions of greatest variance. 1901 - Expectation–Maximization (EM) · Classical Machine Learning
Alternates inferring hidden variables and updating parameters. 1977 - t-SNE · Classical Machine Learning
Maps high-dimensional data to 2D while keeping neighbors close. 2008 - UMAP · Classical Machine Learning
A faster dimensionality reduction that better preserves global structure. 2018 - Perceptron · Deep Learning Foundations
The first artificial neuron: weighted sum followed by a threshold. 1958 - Stochastic Gradient Descent (SGD) · Deep Learning Foundations
Steps downhill using gradients estimated on small mini-batches. 1951 - Backpropagation · Deep Learning Foundations
Computes every parameter's gradient by applying the chain rule backward. 1986 - Convolutional Neural Network (CNN) · Deep Learning Foundations
Slides small learned filters to detect local patterns, composed layer by layer. 1998 - Long Short-Term Memory (LSTM) · Deep Learning Foundations
A gated recurrent network that preserves long-range information. 1997 - Sequence-to-Sequence Learning · Deep Learning Foundations
An encoder reads a sequence and a decoder generates another. 2014 - Word2Vec · Deep Learning Foundations
Learns word vectors by predicting words from context. 2013 - Adam · Deep Learning Foundations
A momentum optimizer with per-parameter adaptive step sizes. 2014 - AdamW (Decoupled Weight Decay) · Deep Learning Foundations
Decouples weight decay from Adam's gradient update. 2017 - Muon Optimizer · Deep Learning Foundations
An optimizer that orthogonalizes momentum updates for matrix parameters. 2024 - Dropout · Deep Learning Foundations
Randomly drops neurons during training to prevent overfitting. 2012 - Batch Normalization · Deep Learning Foundations
Normalizes activations across a batch to stabilize deep training. 2015 - Layer Normalization · Deep Learning Foundations
Normalizes within each example instead of across the batch. 2016 - RMSNorm · Deep Learning Foundations
A simplified LayerNorm that only rescales by root-mean-square. 2019 - Residual Networks (ResNet) · Deep Learning Foundations
Skip connections let signals bypass layers, making very deep nets trainable. 2015 - GELU · Deep Learning Foundations
A smooth activation that gates inputs by their Gaussian probability. 2016 - SwiGLU (Gated Linear Units) · Deep Learning Foundations
Adds a gating branch to the feed-forward layer to improve quality. 2020 - Byte-Pair Encoding (BPE) Tokenization · Transformers & Attention
Builds a subword vocabulary by repeatedly merging the most frequent pairs. 2015 - Attention (Bahdanau) · Transformers & Attention
Lets the decoder look back at the most relevant input positions. 2014 - Transformer & Self-Attention · Transformers & Attention
An attention-only, parallelizable sequence model — the backbone of all LLMs. 2017 - BERT · Transformers & Attention
A bidirectional Transformer encoder pretrained by masked-word prediction. 2018 - T5 (Text-to-Text Transfer Transformer) · Transformers & Attention
Casts every NLP task as text in, text out. 2019 - GPT-3 & In-Context Learning · Transformers & Attention
Large enough language models learn new tasks from a few examples in the prompt. 2020 - Rotary Position Embedding (RoPE) · Transformers & Attention
Encodes position by rotating query/key vectors so attention sees relative distance. 2021 - ALiBi (Attention with Linear Biases) · Transformers & Attention
Replaces position embeddings with a distance-proportional attention penalty. 2021 - Multi-Query Attention (MQA) · Transformers & Attention
All heads share one key/value head, shrinking the KV cache. 2019 - Grouped-Query Attention (GQA) · Transformers & Attention
Groups heads to share K/V, a middle ground between MHA and MQA. 2023 - Multi-head Latent Attention (MLA) · Transformers & Attention
Caches a compressed low-rank latent instead of full keys and values. 2024 - FlashAttention (v1–v3) · Transformers & Attention
Exact attention computed in GPU-memory-aware tiles — faster and lighter. 2022 - Sliding-Window & Sparse Attention · Transformers & Attention
Each token attends to a local window plus a few global tokens for linear cost. 2020 - Native Sparse Attention (NSA) · Transformers & Attention
Trainable, hardware-aligned dynamic sparse attention. 2025 - YaRN (Context Window Extension) · Transformers & Attention
Rescales RoPE frequencies to extend context with little fine-tuning. 2023 - Ring Attention · Transformers & Attention
Shards very long sequences across devices, passing K/V around a ring. 2023 - Linear Attention · Transformers & Attention
Approximates softmax with kernels to make attention linear in length. 2020 - Sparsely-Gated Mixture of Experts · Mixture of Experts
A gating network routes each input to only a few expert sub-networks. 2017 - GShard (Expert Parallelism) · Mixture of Experts
Shards MoE experts across thousands of accelerators. 2020 - Switch Transformer · Mixture of Experts
A simplified MoE that routes each token to a single expert. 2021 - Mixtral (Top-2 MoE) · Mixture of Experts
An open MoE that picks 2 of 8 experts per token. 2024 - DeepSeekMoE (Fine-Grained & Shared Experts) · Mixture of Experts
Finer-grained experts plus always-on shared experts for specialization. 2024 - Auxiliary-Loss-Free Load Balancing · Mixture of Experts
Balances expert load with dynamic biases instead of an extra loss. 2024 - Structured State Spaces (S4) · New Architectures: SSM & Linear Attention
Models very long sequences efficiently with structured state-space equations. 2021 - Mamba (Selective State Spaces) · New Architectures: SSM & Linear Attention
Input-dependent state spaces with linear-time cost and Transformer-like quality. 2023 - Mamba-2 (State Space Duality) · New Architectures: SSM & Linear Attention
Shows SSMs and a class of attention are dual, yielding faster Mamba. 2024 - RWKV · New Architectures: SSM & Linear Attention
Trains in parallel like a Transformer and infers like an RNN. 2023 - Retentive Network (RetNet) · New Architectures: SSM & Linear Attention
An attention alternative with parallel, recurrent and chunkwise modes. 2023 - Hyena Hierarchy · New Architectures: SSM & Linear Attention
Replaces attention with implicitly parameterized long convolutions and gating. 2023 - xLSTM · New Architectures: SSM & Linear Attention
Upgrades LSTM with exponential gating and matrix memory. 2024 - Gated Delta Networks · New Architectures: SSM & Linear Attention
Linear attention combining delta-rule memory updates with gated forgetting. 2024 - Hybrid Architectures (Jamba) · New Architectures: SSM & Linear Attention
Interleaves attention and Mamba layers (plus MoE) in one model. 2024 - Diffusion Language Models (LLaDA) · New Architectures: SSM & Linear Attention
Generates text by iteratively unmasking a fully masked sequence. 2025 - Scaling Laws · Pretraining, Scaling & Distributed Training
Loss falls as a smooth power law in parameters, data and compute. 2020 - Chinchilla Compute-Optimal Scaling · Pretraining, Scaling & Distributed Training
At fixed compute, scale parameters and tokens together (~20 tokens/param). 2022 - Emergent Abilities · Pretraining, Scaling & Distributed Training
Some abilities appear abruptly only past a certain scale. 2022 - μP (Maximal Update Parametrization) · Pretraining, Scaling & Distributed Training
Tune hyperparameters on a small model and transfer them to a large one. 2022 - Multi-Token Prediction (MTP) · Pretraining, Scaling & Distributed Training
Trains the model to predict several future tokens at once. 2024 - Mixed Precision & FP8 Training · Pretraining, Scaling & Distributed Training
Compute in 16- or 8-bit floats while keeping FP32 master weights. 2017 - ZeRO / FSDP (Sharded Data Parallelism) · Pretraining, Scaling & Distributed Training
Shards optimizer states, gradients and parameters across GPUs. 2019 - Tensor Parallelism (Megatron-LM) · Pretraining, Scaling & Distributed Training
Splits a layer's matrix multiplications across GPUs. 2019 - Pipeline Parallelism (GPipe) · Pretraining, Scaling & Distributed Training
Splits layers across devices and streams micro-batches like an assembly line. 2018 - Reinforcement Learning from Human Feedback (RLHF) · Alignment & Preference Optimization
Train a reward model on human preferences, then optimize the policy with RL. 2022 - Reward Modeling (Preference Learning) · Alignment & Preference Optimization
Learns a scoring function from pairwise human comparisons. 2017 - Direct Preference Optimization (DPO) · Alignment & Preference Optimization
Optimizes preferences directly — no reward model or RL loop. 2023 - KTO (Kahneman–Tversky Optimization) · Alignment & Preference Optimization
Aligns from simple good/bad labels instead of pairs. 2024 - ORPO (Odds Ratio Preference Optimization) · Alignment & Preference Optimization
Merges SFT and preference optimization in one step without a reference model. 2024 - SimPO (Simple Preference Optimization) · Alignment & Preference Optimization
Uses length-normalized log-likelihood as an implicit reward, reference-free. 2024 - Constitutional AI / RLAIF · Alignment & Preference Optimization
Uses a written constitution so AI critiques itself; AI feedback replaces human labels. 2022 - Group Relative Policy Optimization (GRPO) · Alignment & Preference Optimization
Samples a group of answers per prompt and uses in-group relative scores instead of a critic. 2024 - RL with Verifiable Rewards (RLVR) · Alignment & Preference Optimization
Rewards come from programmatic checks like math answers or unit tests. 2024 - DAPO · Alignment & Preference Optimization
A set of GRPO fixes and an open system for long-CoT RL. 2025 - Chain-of-Thought Prompting · Reasoning & Test-Time Compute
Prompting the model to write intermediate reasoning before answering. 2022 - Self-Consistency · Reasoning & Test-Time Compute
Samples many reasoning paths and takes the majority answer. 2022 - Tree of Thoughts · Reasoning & Test-Time Compute
Expands reasoning as a tree searched with self-evaluation and backtracking. 2023 - Process Reward Models (PRM) · Reasoning & Test-Time Compute
Scores each reasoning step rather than only the final answer. 2023 - STaR (Self-Taught Reasoner) · Reasoning & Test-Time Compute
Bootstraps reasoning by fine-tuning on the model's own correct rationales. 2022 - Test-Time Compute Scaling · Reasoning & Test-Time Compute
Spending more inference compute — thinking, sampling, search — for better answers. 2024 - DeepSeek-R1 Reasoning RL Recipe · Reasoning & Test-Time Compute
Large-scale outcome-reward RL alone teaches long reasoning and self-reflection. 2025 - Budget Forcing (s1) · Reasoning & Test-Time Compute
Controls test-time compute by forcing thinking to extend or stop. 2025 - Instruction Tuning (FLAN) · Fine-tuning & PEFT
Fine-tuning on many instruction–response tasks for zero-shot generalization. 2021 - Self-Instruct · Fine-tuning & PEFT
The model generates its own instruction data for fine-tuning. 2022 - Adapters · Fine-tuning & PEFT
Inserts small trainable modules between frozen layers. 2019 - Prefix & Prompt Tuning · Fine-tuning & PEFT
Learns only continuous soft-prompt vectors prepended to inputs. 2021 - LoRA (Low-Rank Adaptation) · Fine-tuning & PEFT
Freezes the model and learns weight updates as small low-rank matrices. 2021 - QLoRA · Fine-tuning & PEFT
Fine-tunes LoRA on a 4-bit quantized base, fitting big models on one GPU. 2023 - DoRA (Weight-Decomposed LoRA) · Fine-tuning & PEFT
Splits weights into magnitude and direction, using LoRA for direction. 2024 - Model Merging & Task Arithmetic · Fine-tuning & PEFT
Combines fine-tuned models by adding, subtracting or averaging weights. 2022 - Knowledge Distillation · Quantization, Pruning & Distillation
A small student learns from a large teacher's soft probabilities. 2015 - LLM.int8() & Outlier Handling · Quantization, Pruning & Distillation
8-bit matmul with outlier features kept in 16-bit. 2022 - GPTQ · Quantization, Pruning & Distillation
Second-order, column-by-column post-training weight quantization to 3–4 bits. 2022 - AWQ (Activation-aware Weight Quantization) · Quantization, Pruning & Distillation
Protects the ~1% most salient weights identified by activation magnitude. 2023 - SmoothQuant · Quantization, Pruning & Distillation
Migrates quantization difficulty from activations to weights for W8A8. 2022 - SparseGPT · Quantization, Pruning & Distillation
One-shot pruning of half a large model's weights without retraining. 2023 - BitNet b1.58 (Ternary LLMs) · Quantization, Pruning & Distillation
Trains LLMs from scratch with ternary weights {−1, 0, +1}. 2024 - KV Cache · Inference Acceleration & Serving
Stores past keys and values so each decoding step doesn't recompute them. — - Temperature & Nucleus (Top-p) Sampling · Inference Acceleration & Serving
Controls randomness by sampling only from the top-p probability mass. 2019 - Speculative Decoding · Inference Acceleration & Serving
A small draft model proposes tokens that the big model verifies in one pass. 2022 - Medusa · Inference Acceleration & Serving
Adds extra decoding heads so the model drafts its own speculative tokens. 2024 - EAGLE · Inference Acceleration & Serving
Speculates at the feature level for higher acceptance rates. 2024 - PagedAttention (vLLM) · Inference Acceleration & Serving
Manages KV cache in pages like OS virtual memory. 2023 - Continuous (Iteration-Level) Batching · Inference Acceleration & Serving
Re-forms the batch every decoding step so finished requests leave immediately. 2022 - RadixAttention (Prefix Caching) · Inference Acceleration & Serving
Reuses KV cache across requests sharing prefixes via a radix tree. 2023 - Attention Sinks (StreamingLLM) · Inference Acceleration & Serving
Keeping a few initial sink tokens enables endless streaming generation. 2023 - Prefill–Decode Disaggregation · Inference Acceleration & Serving
Runs prompt prefill and token decoding on separate GPU pools. 2024 - Variational Autoencoder (VAE) · Diffusion, Flow Matching & Generative Models
Encodes data as a distribution in latent space and decodes samples into new data. 2013 - Generative Adversarial Network (GAN) · Diffusion, Flow Matching & Generative Models
A generator and a discriminator improve by competing. 2014 - VQ-VAE (Neural Discrete Representation) · Diffusion, Flow Matching & Generative Models
Encodes images or audio into discrete codebook tokens. 2017 - Denoising Diffusion Probabilistic Models (DDPM) · Diffusion, Flow Matching & Generative Models
Learns to turn noise into images step by step. 2020 - DDIM · Diffusion, Flow Matching & Generative Models
Makes diffusion sampling deterministic, cutting steps from ~1000 to tens. 2020 - Score-Based Models via SDEs · Diffusion, Flow Matching & Generative Models
Unifies diffusion as SDEs; generation solves the reverse-time equation. 2020 - Latent Diffusion Models · Diffusion, Flow Matching & Generative Models
Runs diffusion in a VAE's compressed latent space to cut cost. 2021 - Classifier-Free Guidance (CFG) · Diffusion, Flow Matching & Generative Models
Extrapolates between conditional and unconditional predictions to follow prompts. 2022 - Diffusion Transformer (DiT) · Diffusion, Flow Matching & Generative Models
Replaces the U-Net denoiser with a Transformer. 2022 - Flow Matching · Diffusion, Flow Matching & Generative Models
Learns a velocity field that flows noise into data. 2022 - Rectified Flow · Diffusion, Flow Matching & Generative Models
Straightens generative paths so few or one step suffices. 2022 - Consistency Models · Diffusion, Flow Matching & Generative Models
Maps any point on a trajectory directly to its endpoint for one-step generation. 2023 - ControlNet · Diffusion, Flow Matching & Generative Models
Adds precise spatial control (edges, pose, depth) to diffusion models. 2023 - Visual Autoregressive Modeling (VAR) · Diffusion, Flow Matching & Generative Models
Autoregressive image generation by next-scale rather than next-token prediction. 2024 - Vision Transformer (ViT) · Multimodal & Vision
Treats 16×16 image patches as words for a standard Transformer. 2020 - CLIP (Contrastive Language–Image Pretraining) · Multimodal & Vision
Aligns images and text in one embedding space from 400M pairs. 2021 - SigLIP (Sigmoid Loss) · Multimodal & Vision
Replaces softmax contrastive loss with pairwise sigmoid loss. 2023 - Visual Instruction Tuning (LLaVA) · Multimodal & Vision
Connects a vision encoder to an LLM via a projector, tuned on visual instructions. 2023 - Flamingo · Multimodal & Vision
Inserts cross-attention into a frozen LLM to read interleaved images and text. 2022 - Whisper · Multimodal & Vision
General speech recognition trained on 680K hours of weakly labeled web audio. 2022 - Segment Anything (SAM) · Multimodal & Vision
A promptable foundation model that segments any object from a click or box. 2023 - NeRF & 3D Gaussian Splatting · Multimodal & Vision
Reconstructs 3D scenes renderable from any viewpoint from photos. 2020 - Contrastive Learning (InfoNCE) · RAG & Vector Retrieval
Learns embeddings by pulling positives together and pushing negatives apart. 2018 - BM25 · RAG & Vector Retrieval
Classic keyword ranking by term frequency and rarity. 2009 - Sentence Embeddings (Sentence-BERT) · RAG & Vector Retrieval
Encodes sentences into vectors compared by cosine similarity. 2019 - Dense Passage Retrieval (DPR) · RAG & Vector Retrieval
Dual encoders map questions and passages into one space for retrieval. 2020 - ColBERT (Late Interaction) · RAG & Vector Retrieval
Keeps per-token vectors and matches them late for precise yet fast retrieval. 2020 - HNSW (Hierarchical Navigable Small World) · RAG & Vector Retrieval
A layered small-world graph for millisecond approximate nearest-neighbor search. 2016 - Product Quantization (PQ) · RAG & Vector Retrieval
Splits vectors into chunks, each coded by a small codebook, for compressed fast search. 2011 - FAISS (Billion-scale GPU Similarity Search) · RAG & Vector Retrieval
Meta's library and GPU algorithms for billion-scale vector search. 2017 - Matryoshka Representation Learning · RAG & Vector Retrieval
Embeddings whose prefixes are themselves usable smaller embeddings. 2022 - Retrieval-Augmented Generation (RAG) · RAG & Vector Retrieval
Retrieve relevant documents, then generate grounded answers. 2020 - HyDE (Hypothetical Document Embeddings) · RAG & Vector Retrieval
Generate a hypothetical answer, then retrieve real documents with it. 2022 - GraphRAG · RAG & Vector Retrieval
Builds a knowledge graph and community summaries to answer global questions. 2024 - ReAct (Reason + Act) · Agents & Tool Use
Interleaves reasoning, actions and observations. 2022 - Toolformer · Agents & Tool Use
A model teaches itself when and how to call APIs. 2023 - Function Calling / Tool Use · Agents & Tool Use
Models emit structured function calls that follow a JSON Schema. 2023 - Model Context Protocol (MCP) · Agents & Tool Use
An open protocol connecting AI apps to external tools and data. 2024 - Reflexion · Agents & Tool Use
Agents write verbal self-reflections after failures and reuse them. 2023 - Generative Agents (Memory & Reflection) · Agents & Tool Use
Agents with memory streams, reflection and planning simulate human life. 2023 - Agent–Computer Interface (SWE-agent) · Agents & Tool Use
LM-tailored command interfaces that boost automated bug fixing. 2024 - Q-Learning · Reinforcement Learning
Learns the long-term value of each action in each state. 1992 - Deep Q-Network (DQN) · Reinforcement Learning
Approximates Q with a CNN to play Atari from pixels. 2013 - Policy Gradient (REINFORCE) · Reinforcement Learning
Directly increases the probability of actions that led to high return. 1992 - Proximal Policy Optimization (PPO) · Reinforcement Learning
A stable, simple policy-gradient method that clips update size. 2017 - Soft Actor-Critic (SAC) · Reinforcement Learning
Off-policy actor-critic maximizing both return and entropy. 2018 - AlphaGo / AlphaZero (Self-Play + MCTS) · Reinforcement Learning
Neural evaluation plus tree search, learned from scratch by self-play. 2016 - MuZero · Reinforcement Learning
Plans without knowing the rules by learning a value-focused model. 2019 - World Models · World Models & Self-Supervised Learning
Learn a dreamable environment model, then train the policy inside it. 2018 - Dreamer (Latent Imagination) · World Models & Self-Supervised Learning
Trains behavior by imagining trajectories in a learned latent world model. 2019 - Joint-Embedding Predictive Architecture (JEPA) · World Models & Self-Supervised Learning
Predicts masked content in representation space rather than pixels. 2023 - Genie (Generative Interactive Environments) · World Models & Self-Supervised Learning
Learns playable, controllable worlds from unlabeled videos. 2024 - SimCLR · World Models & Self-Supervised Learning
Learns visual features without labels by matching two augmentations. 2020 - Masked Autoencoders (MAE) · World Models & Self-Supervised Learning
Masks 75% of image patches and reconstructs them to learn representations. 2021 - Graph Neural Networks (GCN / GAT) · Graph Neural Networks & AI for Science
Nodes update by aggregating their neighbors' features. 2016 - AlphaFold 2 · Graph Neural Networks & AI for Science
Predicts protein 3D structure from sequence at near-experimental accuracy. 2021 - AlphaGeometry (Neuro-Symbolic Reasoning) · Graph Neural Networks & AI for Science
An LM proposes constructions; a symbolic engine proves olympiad geometry. 2024 - Feature Attribution (Integrated Gradients / SHAP) · Interpretability
Measures how much each input feature contributed to a prediction. 2017 - Grad-CAM · Interpretability
Gradient-weighted heatmaps showing where a CNN looks. 2016 - Logit Lens & Probing · Interpretability
Decodes intermediate layers into tokens, or probes them with small classifiers. 2020 - Circuits & Induction Heads · Interpretability
Finds circuits implementing algorithms, like induction heads that copy patterns. 2022 - Superposition · Interpretability
Models pack more features than dimensions by superimposing them. 2022 - Sparse Autoencoders (SAE) · Interpretability
Decomposes activations into many sparse, interpretable features. 2023 - Circuit Tracing & Attribution Graphs · Interpretability
Maps causal feature pathways behind a specific response. 2025 - Locating & Editing Facts (ROME) · Interpretability
Locates where facts are stored and edits those weights directly. 2022 - Activation Steering · Interpretability
Adds a direction vector to activations at inference to change behavior. 2023 - Adversarial Examples (FGSM) · Safety, Evaluation & Security
Imperceptible input perturbations that fool models. 2014 - GCG Universal Jailbreak Attack · Safety, Evaluation & Security
Automatically finds gibberish suffixes that jailbreak aligned models. 2023 - Indirect Prompt Injection · Safety, Evaluation & Security
Hides malicious instructions in content the model reads. 2023 - Automated Red Teaming · Safety, Evaluation & Security
Uses one LM to generate test cases that attack another. 2022 - LLM-as-a-Judge · Safety, Evaluation & Security
Strong LLMs grade or compare other models' answers. 2023 - LLM Watermarking · Safety, Evaluation & Security
Biases generation toward a hidden green list so AI text is detectable. 2023 - Differentially Private SGD (DP-SGD) · Safety, Evaluation & Security
Clips per-example gradients and adds noise to protect individuals. 2016 - Weak-to-Strong Generalization · Safety, Evaluation & Security
Supervising strong models with weak-model labels as an analogy for superalignment. 2023 - AI Safety via Debate · Safety, Evaluation & Security
Two AIs debate and a human judges, aiding oversight of hard questions. 2018 - Sleeper Agents (Backdoors that Persist) · Safety, Evaluation & Security
Backdoored models keep hidden behaviors even after safety training. 2024 - Looped Language Models · New Architectures: SSM & Linear Attention
Re-applies the same layers several times, trading depth in parameters for depth in compute. 2018 - Regularized Regression (Lasso / Ridge) · Classical Machine Learning
Penalize linear-model weights to curb overfitting and select features. 1996 - AdaBoost · Classical Machine Learning
Chains weak learners, each focusing on the examples the previous ones got wrong. 1997 - DBSCAN (Density-Based Clustering) · Classical Machine Learning
Clusters by local density; finds arbitrarily shaped clusters and flags noise. 1996 - Naive Bayes · Classical Machine Learning
Applies Bayes’ rule assuming features are independent given the class. 1997 - Gated Recurrent Unit (GRU) · Deep Learning Foundations
A leaner gated RNN than LSTM, with two gates for remembering and forgetting. 2014 - Autoencoder · Deep Learning Foundations
Compress the input to a short code and reconstruct it, learning a compact representation. 2006 - U-Net · Deep Learning Foundations
A U-shaped conv net: downsample, upsample, and skip-connect matching levels. 2015 - Weight Initialization (Xavier / He) · Deep Learning Foundations
Scale initial weights by layer width so signals and gradients neither explode nor vanish. 2010 - GloVe · Deep Learning Foundations
Word vectors trained on global word co-occurrence counts; Word2Vec’s main peer. 2014 - ELMo (Contextual Word Embeddings) · Deep Learning Foundations
Bidirectional LSTM language-model features: the same word gets different vectors in different contexts. 2018 - SentencePiece Tokenization · Transformers & Attention
Trains subword tokenizers on raw text without relying on spaces – works for Chinese and other languages. 2018 - Faster R-CNN (Object Detection) · Multimodal & Vision
Two-stage detector: a network proposes boxes, then classifies and refines them. 2015 - YOLO (Real-Time Object Detection) · Multimodal & Vision
Predicts all boxes and classes in a single forward pass – very fast. 2015 - DETR (Detection Transformer) · Multimodal & Vision
Casts detection as set prediction with a Transformer – no anchors, no NMS. 2020 - Swin Transformer · Multimodal & Vision
Hierarchical vision Transformer with shifted local-window attention. 2021 - DINOv2 (Self-Supervised Visual Features) · World Models & Self-Supervised Learning
Label-free self-distillation on curated images yields general-purpose visual features. 2023 - Actor-Critic & A3C/A2C · Reinforcement Learning
An actor picks actions, a critic scores them; the advantage cuts gradient variance. 2016 - Trust Region Policy Optimization (TRPO) · Reinforcement Learning
Caps each policy update by a KL “trust region” for monotonic improvement. 2015 - DDPG / TD3 (Continuous Control) · Reinforcement Learning
Brings DQN-style learning to continuous actions with a deterministic actor and a critic. 2015 - Conservative Q-Learning (CQL, Offline RL) · Reinforcement Learning
Learns from a fixed dataset by pushing down Q-values of unseen actions. 2020 - Decision Transformer · Reinforcement Learning
RL as sequence modelling: condition on a target return and predict the next action. 2021 - Self-Refine · Reasoning & Test-Time Compute
One model drafts, critiques its own output, then revises – repeated a few times. 2023 - Normalizing Flows (RealNVP / Glow) · Diffusion, Flow Matching & Generative Models
Chains invertible maps from a simple distribution to a complex one, with exact likelihood. 2016 - Connectionist Temporal Classification (CTC) · Multimodal & Vision
Trains speech and handwriting recognizers without frame-level alignment. 2006 - wav2vec 2.0 · Multimodal & Vision
Contrastive self-supervised pre-training on raw audio; strong ASR from little labelled data. 2020 - VALL-E (Neural Codec Language Model TTS) · Multimodal & Vision
Speech as discrete codec tokens generated by a language model – clones a voice from seconds of audio. 2023 - Cross-Encoder Reranking · RAG & Vector Retrieval
Feed query and document together into BERT to score relevance and re-rank candidates. 2019 - Multi-Agent Conversation (AutoGen) · Agents & Tool Use
Multiple specialised LLM agents converse and call tools to solve a task together. 2023 - MemGPT (Hierarchical Agent Memory) · Agents & Tool Use
LLM manages its own memory like an OS – paging between the context window and external storage. 2023 - Membership Inference Attacks · Safety, Evaluation & Security
Tells whether a given record was in a model’s training set – a measure of privacy leakage. 2016
Type or pick a filter to see matching algorithms. Or browse by area below.
0220 areas
Each area lists its algorithms on a timeline with prerequisites and related course units.
Classical Machine Learning18 Statistical learning methods that predate deep learning and still dominate tabular and small-data problems. Principal Component Analysis (PCA) · Logistic Regression · k-Nearest Neighbors Deep Learning Foundations23 How neural nets are trained: backprop, optimizers, normalization, activations and classic architectures. Stochastic Gradient Descent (SGD) · Perceptron · Backpropagation Transformers & Attention18 The backbone of LLMs: attention, positional encodings, tokenization and variants like FlashAttention, GQA/MLA and long context. Attention (Bahdanau) · Byte-Pair Encoding (BPE) Tokenization · Transformer & Self-Attention Mixture of Experts6 Sparse models that activate only some parameters, growing capacity without proportional compute. Sparsely-Gated Mixture of Experts · Switch Transformer · Mixtral (Top-2 MoE) New Architectures: SSM & Linear Attention11 Post-Transformer directions: Mamba, RWKV, linear attention, hybrids and diffusion language models. Mamba (Selective State Spaces) · RWKV · Gated Delta Networks Pretraining, Scaling & Distributed Training9 Scaling laws, hyperparameter transfer, mixed precision and parallelism — how frontier models get trained. Mixed Precision & FP8 Training · Pipeline Parallelism (GPipe) · ZeRO / FSDP (Sharded Data Parallelism) Alignment & Preference Optimization10 Making models helpful and harmless: RLHF, DPO, Constitutional AI, GRPO and RL with verifiable rewards. Reward Modeling (Preference Learning) · Reinforcement Learning from Human Feedback (RLHF) · Constitutional AI / RLAIF Reasoning & Test-Time Compute9 Chain of thought, self-consistency, process rewards and test-time scaling behind o1/R1-style reasoners. Chain-of-Thought Prompting · Self-Consistency · STaR (Self-Taught Reasoner) Fine-tuning & PEFT8 Adapting general models cheaply: LoRA, QLoRA, instruction tuning and model merging. Adapters · Instruction Tuning (FLAN) · LoRA (Low-Rank Adaptation) Quantization, Pruning & Distillation7 Making models smaller, faster and lighter so they run on one GPU, a laptop or a phone. Knowledge Distillation · LLM.int8() & Outlier Handling · GPTQ Inference Acceleration & Serving10 KV cache, speculative decoding, PagedAttention, continuous batching — making LLM serving fast and cheap. Temperature & Nucleus (Top-p) Sampling · Speculative Decoding · Continuous (Iteration-Level) Batching Diffusion, Flow Matching & Generative Models15 VAEs, GANs, diffusion, flow matching and DiT — the algorithms behind image, video and audio generation. Variational Autoencoder (VAE) · VQ-VAE (Neural Discrete Representation) · Denoising Diffusion Probabilistic Models (DDPM) Multimodal & Vision15 ViT, CLIP, VLMs, speech, segmentation and 3D reconstruction — letting AI see and hear. Connectionist Temporal Classification (CTC) · Faster R-CNN (Object Detection) · YOLO (Real-Time Object Detection) RAG & Vector Retrieval13 Embeddings, vector indexes, dense retrieval, reranking and GraphRAG — connecting LLMs to knowledge. BM25 · Product Quantization (PQ) · HNSW (Hierarchical Navigable Small World) Agents & Tool Use9 ReAct, tool calling, MCP, reflection and memory — letting models plan and act. ReAct (Reason + Act) · Function Calling / Tool Use · Reflexion Reinforcement Learning12 From Q-learning to PPO, AlphaZero and MuZero — learning decisions from trial and reward. Q-Learning · Policy Gradient (REINFORCE) · Deep Q-Network (DQN) World Models & Self-Supervised Learning7 Simulating and predicting the world: World Models, Dreamer, JEPA, Genie, plus self-supervised learning like contrastive learning and MAE. World Models · Dreamer (Latent Imagination) · Masked Autoencoders (MAE) Graph Neural Networks & AI for Science3 Graph models like GCN and GAT, plus science systems like AlphaFold and AlphaGeometry. Graph Neural Networks (GCN / GAT) · AlphaFold 2 · AlphaGeometry (Neuro-Symbolic Reasoning) Interpretability9 Opening the black box: attribution, probing, superposition, sparse autoencoders, circuit tracing and steering. Feature Attribution (Integrated Gradients / SHAP) · Logit Lens & Probing · Circuits & Induction Heads Safety, Evaluation & Security11 Adversarial examples, jailbreaks, prompt injection, red teaming, watermarking, differential privacy and scalable oversight. Adversarial Examples (FGSM) · Membership Inference Attacks · Automated Red Teaming
03Learning paths
Five routes through the atlas, in the order the ideas build on each other.
4 steps · 24 algorithms Zero to Deep LearningFrom linear regression all the way to ResNet and the precursors of the Transformer — build intuition for how models are trained. Level 4 steps · 19 algorithms Understanding an LLM End to EndFrom tokenisation to the Transformer, modern architectures, pre-training and inference — every part of an LLM, in nanoGPT order. Level 4 steps · 19 algorithms Post-training & Reasoning ModelsFrom instruction tuning and LoRA to RLHF and DPO, then GRPO, RLVR and DeepSeek-R1 — how ChatGPT and o1/R1 are made. Level 4 steps · 20 algorithms AI Application EngineerFor product engineers: retrieval, RAG, agents, inference serving and security — turn AI into reliable applications fast. Level 5 steps · 26 algorithms Generative Models & Frontier ResearchDiffusion/flow matching, multimodality, new architectures, world models and interpretability — the most active research areas of 2024–2026. Level
04Newest in the atlas
By year of the original paper.
Circuit Tracing & Attribution Graphs Maps causal feature pathways behind a specific response. 2025core DeepSeek-R1 Reasoning RL Recipe Large-scale outcome-reward RL alone teaches long reasoning and self-reflection. 2025core DAPO A set of GRPO fixes and an open system for long-CoT RL. 2025core Diffusion Language Models (LLaDA) Generates text by iteratively unmasking a fully masked sequence. 2025core Native Sparse Attention (NSA) Trainable, hardware-aligned dynamic sparse attention. 2025core Budget Forcing (s1) Controls test-time compute by forcing thinking to extend or stop. 2025ext AlphaGeometry (Neuro-Symbolic Reasoning) An LM proposes constructions; a symbolic engine proves olympiad geometry. 2024core Genie (Generative Interactive Environments) Learns playable, controllable worlds from unlabeled videos. 2024core
How papers are verified
- Year = first public version of the original paper (first arXiv version or publication year).
- 220 algorithms have an original paper; title, year and link were checked against the arXiv API, Crossref or the official page. A green badge means verified.
- 3 entries are not fully verified and carry a yellow badge: Linear Regression、Decision Trees (CART)、KV Cache
- All links go to the paper publisher, arXiv or official docs. For AI: /api/learn/algorithms.json