In Mixture‑of‑Experts (MoE) models, whether experts develop clear domain specializations remains debated. Some studies report domain‑associated expert preferences, while others attribute routing mainly to token identity or syntactic patterns. To reconcile these views, we introduce Routing‑Aware Magnitude Profiling (RAMP), a metric that quantifies domain‑specific computation at the neuron level. Using mixed‑domain calibration data, RAMP discovers neuron sets that overlap 58%‑74% across multi‑source samples of the same domain, but only 15%‑24% across different domains.
We then trace neuron importance throughout pre‑training and post‑training for three fully open MoE families. Although expert‑level importance rankings stay relatively stable, neuron importance continues to reorganize between pre‑training checkpoints; post‑training largely preserves the neuron rankings.
Building on these neuron‑level insights, we propose RAMP‑based Pruning (RAMPr), which selects neurons across experts to retain balanced multi‑domain capabilities. At 75% neuron retention, RAMPr preserves 91%‑98% of the original performance on average. At 50% retention, it reduces GPU memory usage by 36%‑43% and boosts batched prefill throughput by 17%‑39%. The code is released at GitHub.
Review: This work provides a fine‑grained neuron‑level perspective on MoE specialization, revealing that true domain specialization resides in overlapping neuron groups rather than whole experts. The proposed pruning method effectively balances performance and resource consumption, offering a valuable tool for model compression and interpretability research.