Deep neural networks exhibit highly nonlinear dynamics in huge parameter spaces yet show regular macroscopic behavior. We build a statistical‑mechanical description directly in function space, treating parameter configurations as microscopic realizations and functions with their dynamical operators as macroscopic variables. For mean‑squared loss the error evolution is exactly governed by the learning operator $M=JJ^{*}$. Combining the dynamical Boltzmann weight of the conditional stochastic dynamics with the parameter‑space density of states, whose local curvature defines a statistical operator $B$, and integrating over local fluctuations yields
$$ \Phi{\mathrm{fluc}}(M;B)=\frac{\sigma{\xi}^{2}}{2}\log\det(M^{-1}+B)+\mathrm{const}. $$
At fixed spectrum this term is rotationally stationary when $[M,B]=0$; it is minimized by pairing large eigenvalues of $M$ with small eigenvalues of $B$, providing a local restoring contribution against rotational mismatch. For ReLU‑type function spaces under mild stable statistical conditions we have $B=\sigma_{\xi}^{2}L^{*}\mathcal{K}L$, where $L$ measures coarse‑grained second‑order structure. Hence the low‑$B$ sector corresponds (up to bounded anisotropy of $\mathcal{K}$) to low structural curvature, implying a preference for faster relaxation along smooth, data‑adaptive directions. These findings identify function space as a natural macroscopic level for studying stable collective organization in learning.
Review