English

Switch theme

Discover▾

Not sure where to start? This week’s hottest software, models and stacks in three minutes.

AI Apps▾

AI software people use: from assistants like ChatGPT to tools for coding, video and workflows, picked by purpose.

For Agents▾

Software for AI: tools that let your agent (Claude Code, Codex, Cursor…) browse, read docs, run code and remember.

Models▾

Choose models, estimate cost, self-host: prices, rankings, benchmarks, free tiers and compute.

Solutions▾

A goal or an error message? Ready-made stacks, steps and fixes by scenario.

Learn▾

A systematic course from zero to the frontier, with an algorithm library, must-read papers and tutorials.

Frontier▾

Daily AI news, weekly highlights, the latest papers and how AI is changing every discipline.

// Course · stage 4 of 9

Transformers & LLMs

Tokenisation and embeddings, attention, modern LLM architectures, MoE and new architectures, pre-training and scaling laws.

Start the first unit → All stages

5 unitsabout 54 hours47 algorithms

01Units in this stage

In order. Each unit has goals, algorithms, must-read papers, a project and a self-test.

01 · U40 Tokenization & Embeddings BPE tokenisation, per-token pricing, embeddings and semantic similarity. 6 h6 algorithms2 paperslesson ready 02 · U41 Transformers & Attention Self-attention, multi-head attention, positional encoding, residuals and normalisation — build a GPT from scratch. 16 h6 algorithms3 paperslesson ready 03 · U42 Modern LLM Architecture RoPE, SwiGLU, RMSNorm, GQA/MLA, FlashAttention, sliding windows and context extension. 12 h12 algorithms5 papers 04 · U43 MoE & Beyond-Transformer Architectures Sparse experts, load balancing, Mamba/SSMs, linear attention and hybrid architectures, diffusion language models. 10 h18 algorithms5 papers 05 · U44 Pretraining, Data & Scaling Laws Pre-training data pipelines, scaling laws, Chinchilla, μP, MTP and training stability. 10 h6 algorithms4 papers

02Algorithms you will meet

Algorithms Atlas →

Byte-Pair Encoding (BPE) Tokenization Word2Vec Sentence Embeddings (Sentence-BERT)SentencePiece Tokenization GloVe ELMo (Contextual Word Embeddings)Transformer & Self-Attention BERT T5 (Text-to-Text Transfer Transformer)GPT-3 & In-Context Learning Rotary Position Embedding (RoPE)Layer Normalization ALiBi (Attention with Linear Biases)SwiGLU (Gated Linear Units)RMSNorm Multi-Query Attention (MQA)Grouped-Query Attention (GQA)Multi-head Latent Attention (MLA)FlashAttention (v1–v3)Sliding-Window & Sparse Attention Native Sparse Attention (NSA)YaRN (Context Window Extension)Ring Attention Sparsely-Gated Mixture of Experts GShard (Expert Parallelism)Switch Transformer Mixtral (Top-2 MoE)DeepSeekMoE (Fine-Grained & Shared Experts)Auxiliary-Loss-Free Load Balancing Structured State Spaces (S4)Mamba (Selective State Spaces)Mamba-2 (State Space Duality)Linear Attention RWKV Retentive Network (RetNet)Hyena Hierarchy xLSTM Gated Delta Networks Hybrid Architectures (Jamba)Diffusion Language Models (LLaDA)Looped Language Models Scaling Laws Chinchilla Compute-Optimal Scaling Emergent Abilities μP (Maximal Update Parametrization)Multi-Token Prediction (MTP)Mixed Precision & FP8 Training

Part of these routes: Everyday Users · Application Developers · Algorithm Researchers · Product Managers & Leaders · Domain Scientists

Data updated: AI algorithms & curriculum2026-10-22 18:44(UTC+8)