Transformers & LLM Principles
Scaling Laws and Emergence
How loss falls with parameters, data and compute, and what Chinchilla changed.
Advanced 15 min For: Researchers / Developers ✎ In progress
✎ This lesson is still being written. Below is the planned outline; it is not indexed by search engines yet.
What this lesson will cover
- Kaplan et al. power laws
- Compute-optimal training (Chinchilla)
- Emergent abilities and the debate
- Test-time compute scaling