Sparse Mixture‑of‑Experts (MoE) models provide scalable capacity for web services, yet uneven token‑to‑expert routing creates computation and communication bottlenecks. Existing placement and replication techniques redistribute the workload but introduce extra expert copies that compete with token buffers for limited GPU memory. LRMoE is a memory‑aware MoE training system that jointly recalibrates backward demand and balances the retained workload. During the forward pass it plans retention quotas and expert placement under resource constraints while preserving all routed computations. In the backward pass a load‑recalibration metric estimates each token‑expert pair’s contribution to the expert update and selects pairs within the quotas. A capacity‑aware routing algorithm then distributes the retained pairs among eligible instances of their original experts. Physical compaction forms smaller communication payloads and variable‑size grouped GEMMs, reducing backward traffic, computation, and execution‑buffer demand. Experiments on Qwen3‑30B‑A3B and GLM4.5‑106B‑A12B show that LRMoE cuts the cumulative MoE‑layer training time to fixed‑loss targets by 21.2%‑44.8% compared with SYMI.
Review: LRMoE’s demand‑side load recalibration and capacity‑aware routing deliver substantial speed‑ups for MoE training on memory‑constrained nodes, effectively balancing compute, communication, and memory pressures.