Mixture-of-Experts (MoE) architectures have become essential for scaling large language models, and recent work shows that fine-grained expert designs bring clear benefits. Training such models from scratch is costly, so sparsely upcycling from pre‑trained dense models is an attractive alternative. We identify a structural pathology: when fine-grained experts are all derived from a single source model, naive routing collapses and downstream accuracy drops to near‑random. For instance on Qwen3‑1.7B, Drop‑Upcycling‑fine‑grained reaches only 23.2% average accuracy, essentially matching from‑scratch training at 22.2%, while the same method’s coarse‑grained variant stays at 50.2%.
DivMoE is the first framework that achieves structurally balanced routing for fine‑grained MoE upcycling. It introduces two key components: domain‑specialized fine‑grained expert initialization, which first applies domain‑adaptive continual pre‑training to dense models and then extracts experts; and diversity‑constrained routing, a hard constraint that forces each token to activate experts from distinct domain groups, preventing routing collapse. Across two base models and fifteen benchmarks, DivMoE consistently outperforms six upcycling baselines, achieving 55.6% versus the strongest baseline’s 51.6% on Qwen3‑1.7B, and after Stage 2 continual pre‑training it improves over the dense base on every benchmark, closing the regression gap that has plagued prior fine‑grained upcycling. After supervised fine‑tuning on a public reasoning mixture, the 12B‑parameter DivMoE matches Moonlight‑MoE (16B) at 64.5% average accuracy and surpasses a controlled NVIDIA‑Upcycling baseline by 6.1 percentage points.
Review