// ◎ Transformers & Attention
BERT
A bidirectional Transformer encoder pretrained by masked-word prediction.
2018 Foundational core
IN PLAIN WORDS
Mask ~15% of tokens and predict them using both left and right context. It started the pretrain-then-finetune era and still underlies embedding and classification models.
Used in: Search relevance, classification, embedding models
TIMELINE · prerequisites → this → what builds on it
01Original paper
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, et al. · 2018
arXiv ↗ ✓ Verified · arXiv API
02Where it sits
Learn the left column first; the right column builds on this one.
Learn first
Left: prerequisites · right: builds on this · dashed: related. Full map →
03Learning resources
Official docs, explainers, courses and reference code. All links checked.
04Where the course teaches it
On learning paths:Understanding an LLM End to End
For AI / data
- Verification: verified, checked 2026-10-21.
- JSON: /api/learn/algorithms.json (id:
bert)