Skip to content
English

Switch theme

Discover▾

Not sure where to start? This week’s hottest software, models and stacks in three minutes.

AI Apps▾

AI software people use: from assistants like ChatGPT to tools for coding, video and workflows, picked by purpose.

For Agents▾

Software for AI: tools that let your agent (Claude Code, Codex, Cursor…) browse, read docs, run code and remember.

Models▾

Choose models, estimate cost, self-host: prices, rankings, benchmarks, free tiers and compute.

Solutions▾

A goal or an error message? Ready-made stacks, steps and fixes by scenario.

Learn▾

A systematic course from zero to the frontier, with an algorithm library, must-read papers and tutorials.

Frontier▾

Daily AI news, weekly highlights, the latest papers and how AI is changing every discipline.

// Stage 7 · Multimodal & Generative AI

CLIP & Vision-Language Models

Contrastive image–text pre-training, SigLIP, LLaVA-style VLMs and Flamingo.

What you will learn ↓ Code & exercises ↓

about 8 hours4 algorithms4 paperslesson text in progress

01Goals

Do these first: Vision Transformers & Self-Supervised Vision · Instruction Tuning & PEFT

02Algorithms to master

Each card has the original paper, prerequisites and resources.

CLIP (Contrastive Language–Image Pretraining) Aligns images and text in one embedding space from 400M pairs. 2021core SigLIP (Sigmoid Loss) Replaces softmax contrastive loss with pairwise sigmoid loss. 2023core Visual Instruction Tuning (LLaVA) Connects a vision encoder to an LLM via a projector, tuned on visual instructions. 2023core Flamingo Inserts cross-attention into a frozen LLM to read interleaved images and text. 2022ext

03Must-read papers

04Courses & videos

05Hands-on: project, code & exercises

Free, official or author-published pages. Most run in the browser or on a free Colab GPU.

PROJECT

Build a small text-to-image search app with CLIP; then use an open VLM to read 50 charts and score its accuracy.

  1. 1
    mlfoundations/open_clip ↗Reference code

    Do zero-shot image classification and text-to-image search with CLIP.

  2. 2
    github.com ↗Reference code

    Run LLaVA locally and test it on charts and screenshots.

06Self-test

Answer in your head, then open to check.

How does CLIP do zero-shot classification?

Class names are written as text and compared with the image embedding.

What does LLaVA’s projection layer do?

It maps visual features into the language model’s embedding space.

What did SigLIP change from CLIP?

A pairwise sigmoid loss instead of the softmax contrastive loss.

Data updated: AI algorithms & curriculum(UTC+8)