Skip to content
English

Switch theme

Discover▾

Not sure where to start? This week’s hottest software, models and stacks in three minutes.

AI Apps▾

AI software people use: from assistants like ChatGPT to tools for coding, video and workflows, picked by purpose.

For Agents▾

Software for AI: tools that let your agent (Claude Code, Codex, Cursor…) browse, read docs, run code and remember.

Models▾

Choose models, estimate cost, self-host: prices, rankings, benchmarks, free tiers and compute.

Solutions▾

A goal or an error message? Ready-made stacks, steps and fixes by scenario.

Learn▾

A systematic course from zero to the frontier, with an algorithm library, must-read papers and tutorials.

Frontier▾

Daily AI news, weekly highlights, the latest papers and how AI is changing every discipline.

// Stage 7 · Multimodal & Generative AI

Speech, Video & World Models

Whisper, video diffusion (Sora-style DiT), 3-D reconstruction, world models, Dreamer and Genie.

What you will learn ↓ Code & exercises ↓

about 8 hours10 algorithms4 paperslesson text in progress

01Goals

Do these first: Diffusion, Flow Matching & Image Generation · Reinforcement Learning Basics

02Algorithms to master

Each card has the original paper, prerequisites and resources.

Whisper General speech recognition trained on 680K hours of weakly labeled web audio. 2022core Diffusion Transformer (DiT) Replaces the U-Net denoiser with a Transformer. 2022core NeRF & 3D Gaussian Splatting Reconstructs 3D scenes renderable from any viewpoint from photos. 2020core World Models Learn a dreamable environment model, then train the policy inside it. 2018core Dreamer (Latent Imagination) Trains behavior by imagining trajectories in a learned latent world model. 2019core Genie (Generative Interactive Environments) Learns playable, controllable worlds from unlabeled videos. 2024core Joint-Embedding Predictive Architecture (JEPA) Predicts masked content in representation space rather than pixels. 2023core Connectionist Temporal Classification (CTC) Trains speech and handwriting recognizers without frame-level alignment. 2006core wav2vec 2.0 Contrastive self-supervised pre-training on raw audio; strong ASR from little labelled data. 2020core VALL-E (Neural Codec Language Model TTS) Speech as discrete codec tokens generated by a language model – clones a voice from seconds of audio. 2023ext

03Must-read papers

04Courses & videos

05Hands-on: project, code & exercises

Free, official or author-published pages. Most run in the browser or on a free Colab GPU.

PROJECT

Use Whisper to generate time-stamped subtitles for a 10-minute Chinese video and measure the error rate.

  1. 1
    huggingface.co ↗Courses

    Do the ASR and TTS hands-on units of the Hugging Face audio course.

  2. 2
    github.com ↗Reference code

    Transcribe Chinese audio with Whisper and compare error rates by model size.

  3. 3
    Nerfstudio ↗Official docs

    Train your first NeRF / 3D Gaussian scene from a phone video.

06Self-test

Answer in your head, then open to check.

What input features does Whisper use?

Log-Mel spectrograms.

In the Sora technical report, what is a visual “token”?

A spacetime latent patch.

How do world models help reinforcement learning?

Agents can train in imagination, improving sample efficiency.

Data updated: AI algorithms & curriculum(UTC+8)