Skip to content
English

Switch theme

Discover▾

Not sure where to start? This week’s hottest software, models and stacks in three minutes.

AI Apps▾

AI software people use: from assistants like ChatGPT to tools for coding, video and workflows, picked by purpose.

For Agents▾

Software for AI: tools that let your agent (Claude Code, Codex, Cursor…) browse, read docs, run code and remember.

Models▾

Choose models, estimate cost, self-host: prices, rankings, benchmarks, free tiers and compute.

Solutions▾

A goal or an error message? Ready-made stacks, steps and fixes by scenario.

Learn▾

A systematic course from zero to the frontier, with an algorithm library, must-read papers and tutorials.

Frontier▾

Daily AI news, weekly highlights, the latest papers and how AI is changing every discipline.

// Stage 4 · Transformers & LLMs

Modern LLM Architecture

RoPE, SwiGLU, RMSNorm, GQA/MLA, FlashAttention, sliding windows and context extension.

What you will learn ↓ Code & exercises ↓

about 12 hours12 algorithms5 paperslesson text in progress

01Goals

Do these first: Transformers & Attention

02Algorithms to master

Each card has the original paper, prerequisites and resources.

Rotary Position Embedding (RoPE) Encodes position by rotating query/key vectors so attention sees relative distance. 2021core ALiBi (Attention with Linear Biases) Replaces position embeddings with a distance-proportional attention penalty. 2021ext SwiGLU (Gated Linear Units) Adds a gating branch to the feed-forward layer to improve quality. 2020core RMSNorm A simplified LayerNorm that only rescales by root-mean-square. 2019core Multi-Query Attention (MQA) All heads share one key/value head, shrinking the KV cache. 2019ext Grouped-Query Attention (GQA) Groups heads to share K/V, a middle ground between MHA and MQA. 2023core Multi-head Latent Attention (MLA) Caches a compressed low-rank latent instead of full keys and values. 2024core FlashAttention (v1–v3) Exact attention computed in GPU-memory-aware tiles — faster and lighter. 2022core Sliding-Window & Sparse Attention Each token attends to a local window plus a few global tokens for linear cost. 2020core Native Sparse Attention (NSA) Trainable, hardware-aligned dynamic sparse attention. 2025core YaRN (Context Window Extension) Rescales RoPE frequencies to extend context with little fine-tuning. 2023core Ring Attention Shards very long sequences across devices, passing K/V around a ring. 2023core

03Must-read papers

04Courses & videos

05Hands-on: project, code & exercises

Free, official or author-published pages. Most run in the browser or on a free Colab GPU.

PROJECT

Convert your small GPT to RoPE + RMSNorm + SwiGLU + GQA and record changes in parameter count, speed and loss.

  1. 1
    rasbt/LLMs-from-scratch ↗Reference code

    Build GPT with the book, then convert it to a Llama-style model (RoPE, RMSNorm, SwiGLU, GQA) via the bonus chapters.

  2. 2
    scaled_dot_product_attention ↗Official docs

    Swap hand-written attention for PyTorch SDPA and compare speed and memory.

06Self-test

Answer in your head, then open to check.

How much smaller is GQA’s KV cache than MHA’s?

By the ratio of KV heads to query heads.

What does RoPE make attention depend on?

The relative position of the two tokens.

What does FlashAttention mainly save?

Memory reads and writes (IO), not FLOPs.

On these routes: Algorithm Researchers

Data updated: AI algorithms & curriculum(UTC+8)