NeFut Logo NeFut
Admin Login

[CS.AI] ACE: Adaptive Calibration-Free Expert Skipping for MoE-based LLMs

Published at: 2026-09-08 22:00 Last updated: 2026-09-09 09:08
#AI #optimization #LLM

Mixture-of-Experts (MoE) architectures enable efficient scaling of large language models by activating a fixed number of expert slots per token, yet the fixed top‑k routing incurs substantial redundant computation. Existing expert‑skipping approaches usually depend on router confidence, calibration data, or extra training, making it hard to reliably gauge the true contribution of routed experts.\ \ ACE introduces a training‑free, calibration‑free, checkpoint‑preserving framework for token‑adaptive expert skipping in MoE‑based LLMs. It consists of two complementary components: Global Spectral Proxy (GSP), which estimates global transformation capacity from the coupled gate, up, and down projections together with RMSNorm scaling; and Router‑Conditioned Refinement (RCR), which builds expert‑specific direction prototypes from centered router weights and evaluates expert responses along routing‑preferred directions.\ \ During inference, ACE combines the offline GSP and RCR estimates with runtime router gates, skipping an expert slot only when both views identify it as low‑contribution, while always retaining the top‑1 expert. All expert statistics are pre‑computed offline, leaving only table lookups and lightweight scalar operations online.\ \ Extensive experiments across three MoE‑based LLMs and eight benchmarks show that ACE consistently outperforms both static and dynamic baselines, with increasingly pronounced advantages under aggressive skipping. For instance, at a 50% skipping ratio on Qwen3.6‑35B‑A3B, ACE reduces WikiText‑2 perplexity by 7.96% and improves average downstream accuracy by 4.15 percentage points over the strongest competing method.\ \ Review: ACE’s combination of offline spectral analysis and router‑conditioned refinement provides an effective and reliable way to skip experts, offering a practical solution for accelerating inference in large‑scale MoE models.

Original Source: https://arxiv.org/abs/2609.05228

[h] Back to Home