// ◎ Transformers & Attention
ALiBi (Attention with Linear Biases)
Replaces position embeddings with a distance-proportional attention penalty.
ALiBi 线性偏置 2021 Intermediate extended
IN PLAIN WORDS
No position vectors — farther tokens just get a larger penalty on their attention scores, which extrapolates to longer inputs.
Used in: BLOOM, MPT
01Original paper
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
Ofir Press, Noah A. Smith, Mike Lewis · 2021
arXiv ↗ ✓ Verified · arXiv API
02Where it sits
Learn the left column first; the right column builds on this one.
Learn first
ALiBi (Attention with Linear Biases)
Then you can learn
- Nothing in the atlas builds on it yet
Related:Rotary Position Embedding (RoPE)
Left: prerequisites · right: builds on this · dashed: related. Full map →
03Learning resources
Official docs, explainers, courses and reference code. All links checked.
Explainers
04Where the course teaches it
For AI / data
- Verification: verified, checked 2026-10-21.
- JSON: /api/learn/algorithms.json (id:
alibi)