In this paper, we investigate how each component of the Transformer feedforward block architecture design determines the preservation of rank at initialization.
We reinterpret skip connections and normalization, traditionally seen as controlling magnitude, as mechanisms for preserving gradient rank across depth. The very matrix multiplications and nonlinear activations that enhance the network's expressiveness also reduce rank.
We show that skip connections trade off rank collapse against ensemble-like behavior, controlled by the relative scales of the branch and the skip: they route the gradient around the residual branch, where rank is lost, rather than along the long gradient paths that encourage layer composition.
The placement of the normalization layer also controls this tradeoff by establishing the branch-to-skip ratio across depth, unifying much of the literature on normalization placement and depth scaling, particularly explaining why rank collapses for Post-Norm but plateaus for Pre-Norm.
Other architectural aspects, like the two-matrix structure that expands and contracts width, use additional parameters to preserve representation or branch Jacobian rank. The second matrix decorrelates a coherent mean spike that would grow across blocks with a single matrix and uncentered activation, preventing the residual representation from collapsing.
The width expansion between the two matrices maintains full rank of the branch Jacobian: applying the rank-reducing activation in this expanded space leaves enough directions to span the original, at a width that follows Marchenko–Pastur law.
The initialization rank of the input-output Jacobian predicts which networks can train on CIFAR-10.
Together, we recast architecture design for deep networks as navigating an intrinsic tradeoff among rank collapse, ensemble-like behavior, and parameter count.
Blogger's Review: This paper provides an in-depth exploration of the mechanisms for preserving rank in Transformer architectures, emphasizing the role of skip connections and normalization. The findings offer new insights into the training performance of deep learning models, particularly regarding how to optimize network design. Understanding these mechanisms is crucial for developing more efficient deep learning models, especially when dealing with complex datasets.