As model parameters grow, sparsely activated Mixture‑of‑Experts (MoE) has become essential for scaling. Existing models use a hierarchical layout where experts in different layers are topologically independent but store highly similar information, causing redundancy. We replace the conventional layer‑wise Local Expert with a cross‑layer shared Meta Expert. Extensive empirical studies show that models with Meta Experts have lower parameter redundancy, smaller cross‑entropy loss, and higher downstream accuracy. With the same activation count and expert granularity, the model reaches Vanilla MoE performance using only 70%–80% of the experts, greatly improving scaling potential. Theoretical analysis reveals that Meta Experts offer hidden states more flexible residual pathways, enhancing information retrieval efficiency.
Review