Fine‑grained mixture‑of‑experts (MoE) models usually increase the number of experts by narrowing each expert, treating the expert’s shape as a by‑product of routing granularity. LatentMoE moves expert computation into a lower‑dimensional space, making the input dimension $d_{\text{in}}$ and the intermediate width $w$ of each expert explicit design choices. We prove that an expert’s local Jacobian rank satisfies $$\operatorname{rank}(J) \le \min(d_{\text{in}}, w)$@@@MATH_BLOCK3@@@d{\text{in}}$ and $w@@@MATH_BLOCK5@@@d{\text{in}} \approx w$).
We further employ layerwise intrinsic dimension (ID) to guide capacity allocation across depth. Concretely, we let the number of experts $N_{\ell}$ and the shape $(d_{\text{in}}^{\ell}, w^{\ell})$ co‑vary so that each layer’s total parameters, activated FLOPs, and communication volume remain constant. Compared with layer‑homogeneous baselines, the ID‑guided allocation reduces validation loss and improves downstream performance. These findings establish expert shape and layerwise capacity allocation as practical design axes for MoEs.
Review: By making expert geometry an explicit knob and coupling it with layerwise intrinsic dimension, LatentMoE boosts expressive power and downstream results without increasing computational budget, offering a fresh perspective on scaling MoE architectures.