Mixture‑of‑experts (MoE) models already make a sparse choice for each token, deciding which experts should process it. We ask whether a sparse dictionary can reuse that choice. NR‑SAE gives each routed expert a small dictionary and opens only those selected by the frozen router.
At block 15 of Gemma 4 26B‑A4B, after ten million C4 training tokens and across three random seeds, the expert‑aligned autoencoder explains 0.518 of the variance in router‑weighted expert outputs on a fixed evaluation capture. A rank‑addressed control with the same parameters but sixteen times more candidates reaches 0.481, a candidate‑matched control 0.458, and an architecture‑matched control with randomized expert identity 0.387. A transcoder variant also outperforms all three controls: 0.432 versus 0.347, 0.371, and 0.118.
Part of this advantage is architectural: each expert writes into a subspace of at most a quarter of the model width, so naming the expert effectively names the subspace. How much it explains has not been measured. At this layer and budget, the router already knows something a dictionary would otherwise have to learn. Our implementation did not train faster, and semantic interpretability, downstream loss, and causal faithfulness remain untested.
Review