// ◎ Algorithms Atlas

Transformers & Attention

The backbone of LLMs: attention, positional encodings, tokenization and variants like FlashAttention, GQA/MLA and long context.

See all 18 ↓ All areas →

16 core2 extended2014–2025

TIMELINE

2014 Attention (Bah2015 Byte-Pair Enco2017 Transformer & 2018 BERT2018 SentencePiece 2019 T5 (Text-to-Te2019 Multi-Query At2020 GPT-3 & In-Con2020 Sliding-Window2020 Linear Attenti2021 Rotary Positio2021 ALiBi (Attenti2022 FlashAttention2023 Grouped-Query 2023 YaRN (Context 2023 Ring Attention2024 Multi-head Lat2025 Native Sparse

01Core algorithms

Learn these first. Sorted by year.

Attention (Bahdanau) Lets the decoder look back at the most relevant input positions. 2014core Byte-Pair Encoding (BPE) Tokenization Builds a subword vocabulary by repeatedly merging the most frequent pairs. 2015core Transformer & Self-Attention An attention-only, parallelizable sequence model — the backbone of all LLMs. 2017core BERT A bidirectional Transformer encoder pretrained by masked-word prediction. 2018core SentencePiece Tokenization Trains subword tokenizers on raw text without relying on spaces – works for Chinese and other languages. 2018core T5 (Text-to-Text Transfer Transformer) Casts every NLP task as text in, text out. 2019core GPT-3 & In-Context Learning Large enough language models learn new tasks from a few examples in the prompt. 2020core Sliding-Window & Sparse Attention Each token attends to a local window plus a few global tokens for linear cost. 2020core Linear Attention Approximates softmax with kernels to make attention linear in length. 2020core Rotary Position Embedding (RoPE) Encodes position by rotating query/key vectors so attention sees relative distance. 2021core FlashAttention (v1–v3) Exact attention computed in GPU-memory-aware tiles — faster and lighter. 2022core Grouped-Query Attention (GQA) Groups heads to share K/V, a middle ground between MHA and MQA. 2023core YaRN (Context Window Extension) Rescales RoPE frequencies to extend context with little fine-tuning. 2023core Ring Attention Shards very long sequences across devices, passing K/V around a ring. 2023core Multi-head Latent Attention (MLA) Caches a compressed low-rank latent instead of full keys and values. 2024core Native Sparse Attention (NSA) Trainable, hardware-aligned dynamic sparse attention. 2025core

02Extended

Variants and follow-ups worth knowing once the core is clear.

Multi-Query Attention (MQA) All heads share one key/value head, shrinking the KV cache. 2019ext ALiBi (Attention with Linear Biases) Replaces position embeddings with a distance-proportional attention penalty. 2021ext

03Course units that teach these

U00 What AI, ML, Deep Learning and LLMs Are One map of the AI family, and how an LLM “predicts the next token”. 3 h2 algorithms1 paperslesson ready U01 Prompting & Using AI Well Prompting techniques: role, context, examples, step-by-step thinking and structured output. 3 h3 algorithms1 paperslesson ready U32 CNNs, RNNs and the Birth of Attention Convolutional networks, LSTMs, Seq2Seq, word vectors and Bahdanau attention. 10 h7 algorithms5 papers U40 Tokenization & Embeddings BPE tokenisation, per-token pricing, embeddings and semantic similarity. 6 h6 algorithms2 paperslesson ready U41 Transformers & Attention Self-attention, multi-head attention, positional encoding, residuals and normalisation — build a GPT from scratch. 16 h6 algorithms3 paperslesson ready U42 Modern LLM Architecture RoPE, SwiGLU, RMSNorm, GQA/MLA, FlashAttention, sliding windows and context extension. 12 h12 algorithms5 papers