Transformers & LLM Principles
Modern LLM Architectures: Decoders, MoE, Long Context
Decoder-only models, mixture-of-experts, positional encodings and long-context tricks.
Advanced 20 min For: Researchers / Developers ✎ In progress
✎ This lesson is still being written. Below is the planned outline; it is not indexed by search engines yet.
What this lesson will cover
- Decoder-only vs encoder-decoder
- Mixture of experts
- RoPE and context extension
- Efficient attention variants