This paper introduces DeepLoop, which scales sequential computation in Looped Transformers by applying a compact stack of physical blocks, increasing unrolled depth without increasing stored parameters. This reuse alters the residual-scaling problem: in an untied Transformer, each residual branch receives its own parameter update, whereas in a looped Transformer, a shared update aggregates gradients from repeated visits, which are then read back in the next linearized forward pass.
We formalize this tied-depth effect via a first-order perturbation bound controlled by a visit-alignment coefficient $\kappa_R$. The bound recovers the DeepNorm exponent when visits decorrelate, but in the conservative aligned regime, it necessitates an increase in the exponent from $1/4$ to $1/2$ as loop count grows at fixed physical depth.
The resulting method, DeepLoop, maintains the Post-LN DeepNorm architecture with $\alpha=(2N)^{1/2}$ and $\beta=(8N)^{-1/2}$ for unrolled depth $N$. On GPT-style looped language models at GPT-2 small and GPT-2 medium scales, DeepLoop is neutral when no physical block is revisited, but improves validation loss and downstream accuracy once recurrent depth is activated. These results indicate that stable recurrent depth requires residual scaling rules that consider parameter visits, not just nominal layer count.
Blogger's Review: The introduction of DeepLoop offers a fresh perspective on the design of looped transformers, particularly in addressing depth scaling and residual scaling issues. By incorporating a shared update mechanism, DeepLoop effectively enhances model performance, especially in deep recurrent scenarios, showcasing significant advantages. This method warrants further exploration and application for researchers in the deep learning field.