In Mixture-of-Experts (MoE) models, only a sparse subset of experts is activated per token. However, consecutive tokens often activate different experts, leading to frequent weight swapping between slow storage and fast memory on edge devices. Existing solutions are either system-level (caching heuristics) or post-hoc (router fine-tuning), failing to address the root cause during pretraining.
To address this, we propose StickyMoE, a differentiable routing consistency loss that penalizes abrupt expert switches between adjacent tokens, encouraging the router to maintain the same expert assignment across semantically coherent spans. StickyMoE does not require architectural changes, adds a single hyperparameter lambda, and unlike post-hoc methods, allows expert representations and routing decisions to co-adapt from the first training step.
Experiments on small-scale MoE language models show that StickyMoE reduces the expert switch rate by up to 60% with less than 4% perplexity degradation, Pareto-dominating post-hoc fine-tuning on the quality-locality frontier. Routing temporal locality is most efficiently instilled at training time.
Blogger's Review: The introduction of StickyMoE presents a novel approach to tackle the memory efficiency bottleneck in MoE models. By incorporating consistency loss during the training phase, it effectively reduces expert switching, showcasing immense potential in model optimization. The flexibility and efficiency of this method are likely to attract more researchers' attention towards dynamic adjustments in the model training process.