Skip to content
English

Switch theme

Discover▾

Not sure where to start? This week’s hottest software, models and stacks in three minutes.

AI Apps▾

AI software people use: from assistants like ChatGPT to tools for coding, video and workflows, picked by purpose.

For Agents▾

Software for AI: tools that let your agent (Claude Code, Codex, Cursor…) browse, read docs, run code and remember.

Models▾

Choose models, estimate cost, self-host: prices, rankings, benchmarks, free tiers and compute.

Solutions▾

A goal or an error message? Ready-made stacks, steps and fixes by scenario.

Learn▾

A systematic course from zero to the frontier, with an algorithm library, must-read papers and tutorials.

Frontier▾

Daily AI news, weekly highlights, the latest papers and how AI is changing every discipline.

// Stage 4 · Transformers & LLMs

Tokenization & Embeddings

BPE tokenisation, per-token pricing, embeddings and semantic similarity.

Read the lesson → Code & exercises ↓

about 6 hours6 algorithms2 paperslesson text ready

01Goals

Do these first: CNNs, RNNs and the Birth of Attention

02Read on this site

03Algorithms to master

Each card has the original paper, prerequisites and resources.

Byte-Pair Encoding (BPE) Tokenization Builds a subword vocabulary by repeatedly merging the most frequent pairs. 2015core Word2Vec Learns word vectors by predicting words from context. 2013core Sentence Embeddings (Sentence-BERT) Encodes sentences into vectors compared by cosine similarity. 2019core SentencePiece Tokenization Trains subword tokenizers on raw text without relying on spaces – works for Chinese and other languages. 2018core GloVe Word vectors trained on global word co-occurrence counts; Word2Vec’s main peer. 2014ext ELMo (Contextual Word Embeddings) Bidirectional LSTM language-model features: the same word gets different vectors in different contexts. 2018ext

04Must-read papers

05Courses & videos

06Hands-on: project, code & exercises

Free, official or author-published pages. Most run in the browser or on a free Colab GPU.

PROJECT

Following minbpe, train a 4K-vocabulary BPE on a Chinese corpus and compare its token count with tiktoken on the same Chinese text.

  1. 1
    Tiktokenizer ↗Interactive

    Paste Chinese, English and code and count tokens across tokenizers.

  2. 2
    karpathy/minbpe ↗Reference code

    Implement a BPE tokenizer step by step following the repo’s exercise guide.

  3. 3
    google/sentencepiece ↗Reference code

    Train a SentencePiece tokenizer on Chinese text and compare BPE vs Unigram.

07Self-test

Answer in your head, then open to check.

Why does Chinese often use more tokens than English?

Vocabularies contain fewer Chinese merges, so one character can take more than one token.

How does BPE handle unseen words?

It splits them into known sub-words or bytes.

What does it mean when two embeddings are close?

Similar meaning or usage.

On these routes: Everyday Users · Application Developers · Algorithm Researchers

Data updated: AI algorithms & curriculum(UTC+8)