// ◉ Algorithms Atlas
Multimodal & Vision
ViT, CLIP, VLMs, speech, segmentation and 3D reconstruction — letting AI see and hear.
12 core3 extended2006–2023
TIMELINE
01Core algorithms
Learn these first. Sorted by year.
Connectionist Temporal Classification (CTC) Trains speech and handwriting recognizers without frame-level alignment. 2006core Faster R-CNN (Object Detection) Two-stage detector: a network proposes boxes, then classifies and refines them. 2015core YOLO (Real-Time Object Detection) Predicts all boxes and classes in a single forward pass – very fast. 2015core Vision Transformer (ViT) Treats 16×16 image patches as words for a standard Transformer. 2020core NeRF & 3D Gaussian Splatting Reconstructs 3D scenes renderable from any viewpoint from photos. 2020core DETR (Detection Transformer) Casts detection as set prediction with a Transformer – no anchors, no NMS. 2020core wav2vec 2.0 Contrastive self-supervised pre-training on raw audio; strong ASR from little labelled data. 2020core CLIP (Contrastive Language–Image Pretraining) Aligns images and text in one embedding space from 400M pairs. 2021core Whisper General speech recognition trained on 680K hours of weakly labeled web audio. 2022core SigLIP (Sigmoid Loss) Replaces softmax contrastive loss with pairwise sigmoid loss. 2023core Visual Instruction Tuning (LLaVA) Connects a vision encoder to an LLM via a projector, tuned on visual instructions. 2023core Segment Anything (SAM) A promptable foundation model that segments any object from a click or box. 2023core
02Extended
Variants and follow-ups worth knowing once the core is clear.
Swin Transformer Hierarchical vision Transformer with shifted local-window attention. 2021ext Flamingo Inserts cross-attention into a frozen LLM to read interleaved images and text. 2022ext VALL-E (Neural Codec Language Model TTS) Speech as discrete codec tokens generated by a language model – clones a voice from seconds of audio. 2023ext
03Course units that teach these
U70 Vision Transformers & Self-Supervised Vision ViT, SimCLR, MAE, JEPA and SAM. 8 h12 algorithms4 papers U71 CLIP & Vision-Language Models Contrastive image–text pre-training, SigLIP, LLaVA-style VLMs and Flamingo. 8 h4 algorithms4 papers U73 Speech, Video & World Models Whisper, video diffusion (Sora-style DiT), 3-D reconstruction, world models, Dreamer and Genie. 8 h10 algorithms4 papers