// Stage 7 · Multimodal & Generative AI

Diffusion, Flow Matching & Image Generation

VAE/GAN, DDPM/DDIM, score SDEs, latent diffusion, CFG, DiT, flow matching and ControlNet.

What you will learn ↓ Code & exercises ↓

about 14 hours15 algorithms6 paperslesson text in progress

01Goals

Derive DDPM’s training objective (predict the noise)
Understand CFG and guidance strength
Understand how flow matching relates to diffusion

Do these first: Training Deep Nets · Probability, Statistics & Information Theory

02Algorithms to master

Each card has the original paper, prerequisites and resources.

Variational Autoencoder (VAE) Encodes data as a distribution in latent space and decodes samples into new data. 2013core Generative Adversarial Network (GAN) A generator and a discriminator improve by competing. 2014ext VQ-VAE (Neural Discrete Representation) Encodes images or audio into discrete codebook tokens. 2017core Denoising Diffusion Probabilistic Models (DDPM) Learns to turn noise into images step by step. 2020core DDIM Makes diffusion sampling deterministic, cutting steps from ~1000 to tens. 2020ext Score-Based Models via SDEs Unifies diffusion as SDEs; generation solves the reverse-time equation. 2020ext Latent Diffusion Models Runs diffusion in a VAE's compressed latent space to cut cost. 2021core Classifier-Free Guidance (CFG) Extrapolates between conditional and unconditional predictions to follow prompts. 2022core Diffusion Transformer (DiT) Replaces the U-Net denoiser with a Transformer. 2022core Flow Matching Learns a velocity field that flows noise into data. 2022core Rectified Flow Straightens generative paths so few or one step suffices. 2022ext Consistency Models Maps any point on a trajectory directly to its endpoint for one-step generation. 2023ext ControlNet Adds precise spatial control (edges, pose, depth) to diffusion models. 2023core Visual Autoregressive Modeling (VAR) Autoregressive image generation by next-scale rather than next-token prediction. 2024ext Normalizing Flows (RealNVP / Glow) Chains invertible maps from a simple distribution to a complex one, with exact likelihood. 2016ext

03Must-read papers

Auto-Encoding Variational Bayes ↗2013 ✓ verified in Essentials
Denoising Diffusion Probabilistic Models ↗2020 ✓ verified in Essentials
High-Resolution Image Synthesis with Latent Diffusion Models ↗2021 ✓ verified in Essentials
Classifier-Free Diffusion Guidance ↗2022 ✓ verified in Essentials
Scalable Diffusion Models with Transformers ↗2022 ✓ verified in Essentials
Flow Matching for Generative Modeling ↗2022 ✓ verified in Essentials

04Courses & videos

Lilian Weng：What are Diffusion Models? ↗ExplainersEN
huggingface.co ↗CoursesEN
huggingface.co ↗Official docsEN
An introduction to Flow Matching ↗ExplainersEN
Yang Song：Generative Modeling by Estimating Gradients ↗ExplainersEN
Jay Alammar：The Illustrated Stable Diffusion ↗ExplainersEN

05Hands-on: project, code & exercises

Free, official or author-published pages. Most run in the browser or on a free Colab GPU.

PROJECT

Train a small DDPM and a flow-matching model from scratch on MNIST/CIFAR and compare sample quality at equal steps; then do controlled generation with Diffusers + ControlNet.

1
Georgia Tech：Diffusion Explainer ↗Interactive
Watch an image denoise step by step from noise in the interactive demo.
2
huggingface.co ↗Courses
Do the Hugging Face diffusion-course notebooks (train a small diffusion model from scratch).
3
huggingface.co ↗Official docs
Vary steps, CFG scale and scheduler in Diffusers and compare outputs.

06Self-test

Answer in your head, then open to check.

What does the network predict when training DDPM?

The added noise (or an equivalent quantity).

Side effects of too-strong CFG guidance?

Over-saturation and less diversity.

What does flow matching regress?

The velocity field along the probability path.

On these routes: Algorithm Researchers