// ◎ Transformers & Attention

Rotary Position Embedding (RoPE)

Encodes position by rotating query/key vectors so attention sees relative distance.

旋转位置编码 RoPE 2021 Intermediate core

IN PLAIN WORDS

Pair up dimensions and rotate them by position-dependent angles; the dot product then depends only on relative offset. Standard in modern LLMs and the base for context extension like YaRN.

Used in: Llama, Qwen, DeepSeek, Mistral and more

TIMELINE · prerequisites → this → what builds on it

2017 Transformer &…2021 Rotary Position E…2023 YaRN (Context…

01Original paper

RoFormer: Enhanced Transformer with Rotary Position Embedding

Jianlin Su, Yu Lu, Shengfeng Pan, et al. · 2021

arXiv ↗ ✓ Verified · arXiv API

02Where it sits

Learn the left column first; the right column builds on this one.

Learn first

Transformer & Self-Attention

Rotary Position Embedding (RoPE)

Then you can learn

YaRN (Context Window Extension)

Left: prerequisites · right: builds on this · dashed: related. Full map →

03Learning resources

Official docs, explainers, courses and reference code. All links checked.

Explainers

04Where the course teaches it

U41 Transformers & Attention Self-attention, multi-head attention, positional encoding, residuals and normalisation — build a GPT from scratch. 16 h6 algorithms3 paperslesson ready U42 Modern LLM Architecture RoPE, SwiGLU, RMSNorm, GQA/MLA, FlashAttention, sliding windows and context extension. 12 h12 algorithms5 papers

On learning paths:Understanding an LLM End to End

For AI / data

Verification: verified, checked 2026-10-21.
JSON: /api/learn/algorithms.json (id: rope)