Mixture‑of‑Experts (MoE) models reduce inference computation by sparsely activating experts, but the large memory footprint of experts often exceeds GPU capacity, requiring offloading of some experts to the CPU. Existing systems handle GPU‑missed experts either by transferring the entire expert back to the GPU—incurring substantial PCIe latency—or by executing it on the CPU, making the slower CPU computation the bottleneck.
To overcome these limitations, we introduce \method, a CPU‑GPU collaborative MoE inference framework. The key idea is to assign experts adaptively based on their token load: high‑load experts are moved to the GPU to exploit its parallelism, while low‑load experts stay on the CPU to avoid unnecessary data transfer. During inference, the system continuously monitors the progress of both execution paths and dynamically rebalances expert assignments so that the CPU computation path and the GPU transfer‑and‑computation path finish as early as possible, minimizing overall latency.
Extensive experiments on three representative MoE large language models demonstrate that \method consistently yields significant gains across various batch sizes and GPU‑memory budgets: prefill speed improves by up to 51.9% and decode throughput by up to 39.5% over the strongest baseline.
Review