In wide randomly initialized fully‑connected networks, the edge‑of‑chaos condition preserves first‑order input perturbations across depth. However, physics‑informed losses, score matching, and derivative regularization rely on higher‑order input derivatives. For smooth scalar‑input networks we exploit the joint Gaussianity of the finite‑derivative jet that holds in the infinite‑width limit at any fixed depth, and derive mean‑field recursions up to third order. These recursions are exact at the variance fixed point, with finite‑depth corrections that decay geometrically.
At criticality the first‑derivative variance $\mathrm{Var}[\partial_x f]$ remains depth‑invariant, while the second‑derivative variance $\mathrm{Var}[\partial_x^2 f]$ grows linearly whenever the activation has non‑zero curvature: $$\mathrm{Var}[\partial_x^2 f_{\ell}] \sim \ell \cdot \kappa^2,$$ where $\kappa$ denotes the mean curvature of the activation function. The resulting third‑order system closes on mean‑field susceptibilities, yielding a solvable closed recursion.
For residual networks with branch scale $L^{-1/2}$, under explicit regularity assumptions we prove that the variance of any fixed finite derivative order is uniformly bounded across all depths: $$\sup_{\ell}\,\mathrm{Var}[\partial_x^k f_{\ell}] < \infty,\quad k\in\mathbb{N}.$$ Simulations confirm the critical growth laws, the residual bound, and the closed recursion. These results pertain to initialization only and do not directly predict the performance of trained networks.
Review