Sparse Mixture-of-Experts (MoE) models rely heavily on expert activation patterns for inference efficiency. Speculative decoding (SD) accelerates autoregressive generation by verifying multiple draft tokens in parallel, but existing draft selection strategies mainly optimize acceptance likelihood. In large-scale MoE models, selecting draft tokens also determines the union of experts activated during verification.
We observe that confidence-driven SD can introduce \textit{expert scattering}: high-probability draft tokens may route to disjoint experts, increasing expert-weight memory traffic and reducing speedup from speculation. Motivated by this, we revisit draft-tree selection under the non-uniform memory-cost structure of MoE inference.
We propose \textsc{EcoSpec}, a cost-aware speculative decoding framework that incorporates predicted marginal expert activation cost into draft selection. With a lightweight expert predictor and a dynamic expert buffer, \textsc{EcoSpec} favors draft paths that preserve high acceptance likelihood while reusing experts already covered by the current verification set, without modifying the target-model verification rule.
We evaluate \textsc{EcoSpec} on three large-scale MoE models, including DeepSeek-V3.1 (671B), Qwen3-235B-A22B, and GPT-OSS-120B, across reasoning, coding, question-answering, and dialogue benchmarks. \textsc{EcoSpec} consistently reduces active expert footprints and improves end-to-end decoding speed, achieving up to $1.62 \times$ speedup. These results highlight the importance of accounting for expert activation cost for efficient speculative decoding in large-scale MoE models.
Blogger's Review: This research significantly enhances the inference efficiency of mixture-of-experts models by introducing cost-aware speculative decoding, addressing the issue of expert scattering caused by high-probability draft tokens. It underscores the importance of optimizing expert activation in large-scale language models. Future work could further explore similar optimizations in more complex models.