Abstract
Sparse Mixture-of-Experts (MoE) models expand foundation model capacity through conditional expert activation, but deploying full expert pools remains challenging under limited accelerator memory. While expert offloading alleviates memory pressure by moving inactive experts to host memory or storage, it introduces a routing-dependent transfer bottleneck: required experts are only known after native top-K routing, serializing routing, expert loading, and execution during inference.
To address this bottleneck, we propose SpecPrefetch, a parameter-efficient prefetching framework for offloaded MoE inference. SpecPrefetch uses a shared lightweight adapter to predict next-layer expert candidates solely for asynchronous transfer, while the frozen native router still determines the final executed experts. By separating transfer prediction from execution routing, SpecPrefetch reduces exposed expert-loading latency without altering pretrained routing semantics, so prediction errors impact transfer efficiency rather than model outputs.
Additionally, a window-aware scheduler prioritizes feasible transfers under cache and bandwidth constraints. Across Qwen3-VL-30B-A3B and DeepSeek-VL2-Tiny, SpecPrefetch achieves the best average expert recall in 9 out of 10 model-benchmark settings with substantially fewer trainable parameters than learned predictor baselines. On a Snapdragon 8 Elite device, SpecPrefetch further improves decoding throughput by up to 20% over a compute-optimized offloading runtime, demonstrating practical benefits for storage-constrained MoE deployment. The code and model weights are available at GitHub.
Blogger's Review: The design of SpecPrefetch significantly enhances the inference efficiency of sparse MoE models by decoupling the loading of experts from routing. This innovation not only optimizes the speed of model access but also reduces the demand for computational resources, showcasing the potential for future advancements in MoE model development.