NeFut Logo NeFut
Admin Login

[CS.AI] From Concentration to Differentiation and Back: Routing Effective Rank in MoE Reasoning Cohorts

Published at: 2026-09-10 22:00 Last updated: 2026-09-12 06:35
#AI #Machine Learning #LLM

During test‑time inference, a model generates multiple rollouts that form cohorts, yet there is no label‑free way to describe how their internal computation reorganizes as inference proceeds. We introduce routing effective rank ($d_{\text{eff}}$), the entropy‑effective dimensionality of a cross‑rollout graph built from MoE expert‑routing similarity.\

Across ten MoE configurations and five math/science benchmarks, $d_{\text{eff}}$ follows a reproducible low‑high‑low trajectory: in 98.5% of 3,105 model‑question cohorts an interior maximum appears at intermediate token budgets, indicating early concentration of routing similarity, maximal differentiation in the middle, and reconcentration later. The timing of this maximum varies systematically with architecture and reasoning effort.\

An exact decomposition separates the trajectory into common‑mode mass and residual spectral dimensionality. Reallocation of the common mode accounts for roughly two‑thirds of the trajectory, while the residual spectrum contributes about one‑quarter and retains substantial variation beyond the common mode.\

Among non‑unanimous cohorts, increases in common‑mode concentration strongly predict same‑answer recoverability. Higher reasoning effort delays the peak by $2.59$ octaves (approximately a six‑fold increase in token budget) and consistently expands the high‑rank period across all four tested architectures—placing the effort effect in timing and duration rather than peak amplitude.\

By separating structural monitoring from answer selection, routing effective rank serves as a decomposable, label‑free diagnostic of cohort organization, offering a principled spectral lens on how MoE reasoning cohorts differentiate and reconcentrate over inference time.\

Review

Original Source: https://arxiv.org/abs/2609.06403

[h] Back to Home