Skip to content
English

Switch theme

Discover▾

Not sure where to start? This week’s hottest software, models and stacks in three minutes.

AI Apps▾

AI software people use: from assistants like ChatGPT to tools for coding, video and workflows, picked by purpose.

For Agents▾

Software for AI: tools that let your agent (Claude Code, Codex, Cursor…) browse, read docs, run code and remember.

Models▾

Choose models, estimate cost, self-host: prices, rankings, benchmarks, free tiers and compute.

Solutions▾

A goal or an error message? Ready-made stacks, steps and fixes by scenario.

Learn▾

A systematic course from zero to the frontier, with an algorithm library, must-read papers and tutorials.

Frontier▾

Daily AI news, weekly highlights, the latest papers and how AI is changing every discipline.

// Stage 4 · Transformers & LLMs

Pretraining, Data & Scaling Laws

Pre-training data pipelines, scaling laws, Chinchilla, μP, MTP and training stability.

What you will learn ↓ Code & exercises ↓

about 10 hours6 algorithms4 paperslesson text in progress

01Goals

Do these first: Transformers & Attention

02Algorithms to master

Each card has the original paper, prerequisites and resources.

Scaling Laws Loss falls as a smooth power law in parameters, data and compute. 2020core Chinchilla Compute-Optimal Scaling At fixed compute, scale parameters and tokens together (~20 tokens/param). 2022core Emergent Abilities Some abilities appear abruptly only past a certain scale. 2022core μP (Maximal Update Parametrization) Tune hyperparameters on a small model and transfer them to a large one. 2022ext Multi-Token Prediction (MTP) Trains the model to predict several future tokens at once. 2024core Mixed Precision & FP8 Training Compute in 16- or 8-bit floats while keeping FP32 master weights. 2017core

03Must-read papers

04Courses & videos

05Hands-on: project, code & exercises

Free, official or author-published pages. Most run in the browser or on a free Colab GPU.

PROJECT

Train small GPTs of different sizes (1M–20M parameters) to convergence and fit your own scaling curve.

  1. 1
    github.com ↗Reference code

    Run the full tokenise → pretrain → fine-tune → chat UI pipeline (needs a rented GPU).

  2. 2
    karpathy/llm.c ↗Reference code

    Read the C/CUDA GPT-2 training code to see every step of an iteration.

  3. 3
    github.com ↗Reference code

    Understand data, tensor, pipeline and context parallelism from minimal code.

06Self-test

Answer in your head, then open to check.

Chinchilla’s rule-of-thumb ratio?

About 20 training tokens per parameter.

Rough formula for training FLOPs?

About 6 × parameters × tokens.

Why are many models “over-trained” beyond the Chinchilla ratio?

Extra training buys a smaller, cheaper model to serve.

On these routes: Algorithm Researchers · Product Managers & Leaders

Data updated: AI algorithms & curriculum(UTC+8)