Neural networks trained beyond the memorization stage often undergo a delayed transition to generalization, a phenomenon known as grokking. While theory has begun to explain why this transition occurs, the quantitative description of when it happens across hyperparameter space remains missing. We systematically explored 384 configurations of two‑hidden‑layer MLPs on a modular arithmetic task, charted the memorization‑to‑generalization boundary, and fitted a power‑law scaling for the onset time of generalization:
$$T_{\mathrm{grok}} \propto H^{-0.27}\, D^{-2.04}\, \eta^{-0.50}\, \lambda^{-0.64}$$
Here $H$ denotes hidden‑layer width, $D$ the data complexity (training sample count), $\eta$ the learning rate, and $\lambda$ the weight‑decay coefficient. The fit yields $R^2=0.732$, which rises to $0.821$ when interaction terms are included. The hierarchy of exponents shows that data complexity $D^{-2.04}$ dominates the regime shift, while model capacity $H^{-0.27}$ plays a secondary role: doubling the data accelerates generalization by roughly $4\times$, whereas doubling the width yields only about $1.2\times$ speed‑up.
A sharp phase boundary emerges at $\lambda \gtrsim 1.0$, cleanly separating grokking from non‑grokking configurations. Weight‑norm trajectories exhibit monotonic compression during the transition, consistent with implicit regularization steering the network toward low‑complexity solutions.
These findings provide a quantitative foundation for predicting and controlling memorization‑to‑generalization transitions in over‑parameterized networks, suggesting that adjusting data scale and regularization strength can effectively manage learning dynamics.
Review