Skip to content
English

Switch theme

Discover▾

Not sure where to start? This week’s hottest software, models and stacks in three minutes.

AI Apps▾

AI software people use: from assistants like ChatGPT to tools for coding, video and workflows, picked by purpose.

For Agents▾

Software for AI: tools that let your agent (Claude Code, Codex, Cursor…) browse, read docs, run code and remember.

Models▾

Choose models, estimate cost, self-host: prices, rankings, benchmarks, free tiers and compute.

Solutions▾

A goal or an error message? Ready-made stacks, steps and fixes by scenario.

Learn▾

A systematic course from zero to the frontier, with an algorithm library, must-read papers and tutorials.

Frontier▾

Daily AI news, weekly highlights, the latest papers and how AI is changing every discipline.

// Stage 6 · Inference, Deployment & Efficiency

GPUs, Distributed Training & AI Chips

Memory, bandwidth and compute; mixed precision; data/tensor/pipeline parallelism; ZeRO/FSDP; the chip landscape.

What you will learn ↓ Code & exercises ↓

about 10 hours6 algorithms4 paperslesson text in progress

01Goals

Do these first: Pretraining, Data & Scaling Laws

02Algorithms to master

Each card has the original paper, prerequisites and resources.

Mixed Precision & FP8 Training Compute in 16- or 8-bit floats while keeping FP32 master weights. 2017core ZeRO / FSDP (Sharded Data Parallelism) Shards optimizer states, gradients and parameters across GPUs. 2019core Tensor Parallelism (Megatron-LM) Splits a layer's matrix multiplications across GPUs. 2019core Pipeline Parallelism (GPipe) Splits layers across devices and streams micro-batches like an assembly line. 2018core Ring Attention Shards very long sequences across devices, passing K/V around a ring. 2023core GShard (Expert Parallelism) Shards MoE experts across thousands of accelerators. 2020ext

03Must-read papers

04Courses & videos

05Hands-on: project, code & exercises

Free, official or author-published pages. Most run in the browser or on a free Colab GPU.

PROJECT

Train a small model with FSDP on 2 GPUs (or simulated on Colab) and compare memory use with DDP.

  1. 1
    srush/GPU-Puzzles ↗Exercises

    Solve 14 CUDA puzzles in a notebook to grasp threads, blocks and shared memory.

  2. 2
    triton-lang.org ↗Official docs

    Write Triton kernels for vector add, softmax and matmul with the official tutorials.

  3. 3
    github.com ↗Reference code

    Run data and tensor parallelism on several GPUs and compare communication.

06Self-test

Answer in your head, then open to check.

Roughly how many bytes per parameter does mixed-precision AdamW training take?

About 16 bytes (weights, gradients, two moments and master weights).

Is tensor parallelism for across machines or within one?

Within a machine, over fast interconnect.

What does ZeRO-3 shard?

Parameters, gradients and optimiser states.

On these routes: Algorithm Researchers

Data updated: AI algorithms & curriculum(UTC+8)