Skip to content
English

Switch theme

Discover▾

Not sure where to start? This week’s hottest software, models and stacks in three minutes.

AI Apps▾

AI software people use: from assistants like ChatGPT to tools for coding, video and workflows, picked by purpose.

For Agents▾

Software for AI: tools that let your agent (Claude Code, Codex, Cursor…) browse, read docs, run code and remember.

Models▾

Choose models, estimate cost, self-host: prices, rankings, benchmarks, free tiers and compute.

Solutions▾

A goal or an error message? Ready-made stacks, steps and fixes by scenario.

Learn▾

A systematic course from zero to the frontier, with an algorithm library, must-read papers and tutorials.

Frontier▾

Daily AI news, weekly highlights, the latest papers and how AI is changing every discipline.

// Stage 4 · Transformers & LLMs

Transformers & Attention

Self-attention, multi-head attention, positional encoding, residuals and normalisation — build a GPT from scratch.

Read the lesson → Code & exercises ↓

about 16 hours6 algorithms3 paperslesson text ready

01Goals

Do these first: Tokenization & Embeddings · Training Deep Nets

02Read on this site

03Algorithms to master

Each card has the original paper, prerequisites and resources.

Transformer & Self-Attention An attention-only, parallelizable sequence model — the backbone of all LLMs. 2017core BERT A bidirectional Transformer encoder pretrained by masked-word prediction. 2018core T5 (Text-to-Text Transfer Transformer) Casts every NLP task as text in, text out. 2019core GPT-3 & In-Context Learning Large enough language models learn new tasks from a few examples in the prompt. 2020core Rotary Position Embedding (RoPE) Encodes position by rotating query/key vectors so attention sees relative distance. 2021core Layer Normalization Normalizes within each example instead of across the batch. 2016core

04Must-read papers

05Courses & videos

06Hands-on: project, code & exercises

Free, official or author-published pages. Most run in the browser or on a free Colab GPU.

PROJECT

Following nanoGPT, train a character-level GPT on Shakespeare or classical Chinese poetry and visualise one layer’s attention.

  1. 1
    Georgia Tech：Transformer Explainer ↗Interactive

    Type a sentence and step through GPT-2’s attention and next-token probabilities.

  2. 2
    bbycroft ↗Interactive

    Watch one forward pass end to end in 3D.

  3. 3
    github.com ↗Reference code

    Train a character-level GPT from scratch on the Shakespeare dataset.

07Self-test

Answer in your head, then open to check.

Why divide attention scores by √d?

To stop large dot products from saturating the softmax.

What does the causal mask do?

It lets each position see only the tokens before it.

Why multi-head rather than single-head attention?

Different heads can attend to different relationship sub-spaces.

On these routes: Application Developers · Algorithm Researchers · Domain Scientists

Data updated: AI algorithms & curriculum(UTC+8)