Skip to content
English

Switch theme

Discover▾

Not sure where to start? This week’s hottest software, models and stacks in three minutes.

AI Apps▾

AI software people use: from assistants like ChatGPT to tools for coding, video and workflows, picked by purpose.

For Agents▾

Software for AI: tools that let your agent (Claude Code, Codex, Cursor…) browse, read docs, run code and remember.

Models▾

Choose models, estimate cost, self-host: prices, rankings, benchmarks, free tiers and compute.

Solutions▾

A goal or an error message? Ready-made stacks, steps and fixes by scenario.

Learn▾

A systematic course from zero to the frontier, with an algorithm library, must-read papers and tutorials.

Frontier▾

Daily AI news, weekly highlights, the latest papers and how AI is changing every discipline.

// Stage 9 · Safety, Evaluation & Frontier

Evaluation & Benchmarks

Major benchmarks, arenas, data contamination and the limits of capability evaluation.

Read the lesson → Code & exercises ↓

about 6 hours2 algorithms2 paperslesson text ready

01Goals

Do these first: Pretraining, Data & Scaling Laws

02Read on this site

03Algorithms to master

Each card has the original paper, prerequisites and resources.

LLM-as-a-Judge Strong LLMs grade or compare other models' answers. 2023core Emergent Abilities Some abilities appear abruptly only past a certain scale. 2022core

04Must-read papers

05Courses & videos

06Hands-on: project, code & exercises

Free, official or author-published pages. Most run in the browser or on a free Colab GPU.

PROJECT

Run the same set of benchmarks on 3 open models with lm-evaluation-harness and compare with the officially reported numbers.

  1. 1
    EleutherAI lm-evaluation-harness ↗Reference code

    Evaluate an open model on MMLU, GSM8K and more with the harness.

  2. 2
    github.com ↗Reference code

    Run a ready-made Inspect eval, then adapt it to your own task.

  3. 3
    LMArena ↗Tools

    Vote in blind battles to see how human-preference evals work and where they fail.

07Self-test

Answer in your head, then open to check.

What is data contamination?

Test questions appearing in the training data.

What model do arena rankings use?

Bradley–Terry / Elo-style pairwise comparison models.

Limits of a single benchmark score?

Narrow coverage and easy to optimise for.

On these routes: Application Developers · Product Managers & Leaders

Data updated: AI algorithms & curriculum(UTC+8)