// ◎ Transformers & Attention

Ring Attention

Shards very long sequences across devices, passing K/V around a ring.

环形注意力 Ring Attention 2023 Research frontier core

IN PLAIN WORDS

Each device holds a chunk and computes attention with local K/V while passing K/V to the next device, overlapping communication. Context scales with device count — a form of context parallelism.

Used in: Million-token context training

TIMELINE · prerequisites → this → what builds on it

2022 FlashAttentio…2023 Ring Attention

01Original paper

Ring Attention with Blockwise Transformers for Near-Infinite Context

Hao Liu, Matei Zaharia, Pieter Abbeel · 2023

arXiv ↗ ✓ Verified · arXiv API

02Where it sits

Learn the left column first; the right column builds on this one.

Learn first

FlashAttention (v1–v3)

Ring Attention

Then you can learn

Nothing in the atlas builds on it yet

Left: prerequisites · right: builds on this · dashed: related. Full map →

03Learning resources

Official docs, explainers, courses and reference code. All links checked.

Reference code

github.com ↗EN

Books

Hugging Face：The Ultra-Scale Playbook ↗EN

04Where the course teaches it

U42 Modern LLM Architecture RoPE, SwiGLU, RMSNorm, GQA/MLA, FlashAttention, sliding windows and context extension. 12 h12 algorithms5 papers U64 GPUs, Distributed Training & AI Chips Memory, bandwidth and compute; mixed precision; data/tensor/pipeline parallelism; ZeRO/FSDP; the chip landscape. 10 h6 algorithms4 papers

For AI / data

Verification: verified, checked 2026-10-21.
JSON: /api/learn/algorithms.json (id: ring-attention)