In the service of Mixture-of-Experts (MoE) models, expert parallelism has emerged as the prevailing paradigm, with its efficiency reliant on the communication and computation latencies of GPUs, directly linked to the placement of experts on these GPUs. Existing optimization methods for expert placement primarily leverage past request patterns; however, they exhibit deficiencies when confronted with diverse and rapidly changing request patterns, highlighting the need for an online, proactive approach.
Implementing such an approach entails addressing several challenges: the uncertainty tied to the expert activation of incoming requests, the cost of expert migration, and the NP-hard complexity inherent in optimization. Hence, we introduce Director, a novel distributed MoE serving system designed to minimize end-to-end latency through prediction-driven, online expert placement.
Director employs either a lightweight cascaded predictor or a low-bit quantized replica to predict the expert activation patterns of incoming requests. An online migration module subsequently enacts these changes with near-zero downtime by executing migrations during compute-bound phases, keeping disruptions bounded. At its core, a relaxation-based expert placement optimizer operates under capacity constraints, runs in polynomial time, and achieves a $(1+\epsilon)$ approximation ratio. Finally, we implemented a prototype and demonstrated, through extensive experiments, an end-to-end latency reduction of $11\sim55\%$ for popular MoE models (e.g., Mistral, DeepSeek, and Qwen) compared to existing methods.
Blogger's Review: The Director system effectively addresses the challenges posed by dynamic requests through proactive expert placement, showcasing a novel strategy for enhancing efficiency in MoE model serving. Its prediction-based approach not only improves performance but also maintains system stability, making it a noteworthy development.