// ⬡ Deep Learning Foundations
SwiGLU (Gated Linear Units)
Adds a gating branch to the feed-forward layer to improve quality.
SwiGLU 2020 Intermediate core
IN PLAIN WORDS
The FFN projects the input twice; one goes through Swish as a gate multiplying the other. Better at equal parameters and now standard.
Used in: PaLM, Llama, Qwen and others
TIMELINE · prerequisites → this → what builds on it
01Original paper
02Where it sits
Learn the left column first; the right column builds on this one.
Learn first
SwiGLU (Gated Linear Units)
Then you can learn
- Nothing in the atlas builds on it yet
Related:RMSNorm
Left: prerequisites · right: builds on this · dashed: related. Full map →
03Learning resources
Official docs, explainers, courses and reference code. All links checked.
Reference code
04Where the course teaches it
On learning paths:Understanding an LLM End to End
For AI / data
- Verification: verified, checked 2026-10-21.
- JSON: /api/learn/algorithms.json (id:
swiglu)