In this new study, the Muon optimizer reaches the grokking threshold on modular arithmetic faster than AdamW. Previous research attributed this phenomenon to "spectral-norm constraints plus orthogonalized momentum," but did not isolate the specific mechanism. To gain deeper insights into Muon's behavior, we conducted multi-seed and multi-learning-rate experiments to decompose and stress-test its effects.
First, an ablation study shows that the speedup comes from orthogonalization (the Newton-Schulz iteration): orthogonalization-only matches the full Muon speed, while spectral-only is no faster than AdamW and is unreliable, a verdict that holds across learning rates.
Second, a mechanistic analysis finds that orthogonalizing optimizers achieve generalization at roughly 3x lower spectral norm and, when controlling for how much the embedding actually moves, settle into a lower-norm solution rather than merely perturbing the embedding less.
Third, reducing the Newton-Schulz iteration count from five to one accelerates reaching the threshold but makes the grokked solution fragile, prone to transient collapse, with fragility that grows with learning rate; a single iteration is fast and stable only at small learning rates, while the canonical five iterations are the robust choice against learning rates.
Moreover, we demonstrate that spectral scaling can be dropped at no measured cost. A methodological thread runs throughout: under a stability-aware metric, "faster" claims about grokking optimizers can invert, thus we report both first-crossing and sustained-grok times. To support reproducibility, we release our full training and analysis code at GitHub.
Blogger's Review: This paper delves into the internal mechanisms of the Muon optimizer, particularly highlighting the importance of orthogonalization for accelerating grokking. The experiments demonstrate that the design of the optimizer not only affects speed but also directly impacts the stability of the solution, providing significant insights for future research. It is hoped that more studies will combine theory and practice in the design of optimizers.