Specter: Fast MoE Inference via Quantization-Enabled Multi-Step Speculative Offloading

One-line takeaway

Specter achieves up to 1.88x TPOT speedup via quantized speculative offloading for memory-constrained MoE inference on single GPUs.

Specter通过量化与投机解码结合，实现专家模型的高效离载推理，在单请求场景下平均提速1.5倍，显著提升内存受限GPU上MoE模型的推理效率。

Key points

Specter introduces a quantization-enabled speculative offloading framework for MoE models, combining lightweight draft models with multi-step lookahead to predict expert usage. This enables efficient expert movement and execution planning ahead of verification, reducing I/O bottlenecks. It achieves up to 1.88x speedup in token throughput on single-GPU systems with limited memory.

• Specter uses quantized draft models for speculative decoding to predict future expert activations.
• It enables proactive expert offloading and execution planning, reducing PCIe I/O bottlenecks.
• Achieves up to 1.88x speedup in time per output token (TPOT) over state-of-the-art on single-GPU MoE inference.
• Makes memory-constrained MoE models practical on commodity hardware via efficient offloading.

Written only from the official arXiv abstract by an open-source model, then checked automatically (every number and name must appear in the abstract). The paper is authoritative.