Stage 3 · Building with AI
Multimodal AI
Models that see, hear, speak and generate images and video.
3 lessons
- 01 How Vision-Language Models See Image encoders, patches, projection layers and multimodal training. Advanced 20 min ✎ In progress
- 02 Diffusion Models for Images and Video Noise, denoising, latent diffusion, guidance and video models. Advanced 20 min ✎ In progress
- 03 Speech Recognition and Speech Synthesis How Whisper-style ASR and modern TTS work, and real-time voice agents. Intermediate 15 min ✎ In progress
Lessons here are still being written — start with these course units
Each unit has goals, algorithms, must-read papers, a project and a self-test.
U71 CLIP & Vision-Language Models Contrastive image–text pre-training, SigLIP, LLaVA-style VLMs and Flamingo. 8 h4 algorithms4 papers U72 Diffusion, Flow Matching & Image Generation VAE/GAN, DDPM/DDIM, score SDEs, latent diffusion, CFG, DiT, flow matching and ControlNet. 14 h15 algorithms6 papers U73 Speech, Video & World Models Whisper, video diffusion (Sora-style DiT), 3-D reconstruction, world models, Dreamer and Genie. 8 h10 algorithms4 papers