We introduce gate-zero growth, a function-preserving (FP) operator for continual learning that adds new residual blocks through a zero-initialised gate. Under a transversality condition, gate-zero growth induces rank separation in the functional Jacobian: old directions remain unchanged, new-weight directions are exactly flat at the growth point, and new gate directions are the only first-order source of new functional variation.
As gates open during continual learning, function drift is $O(\|\boldsymbol{\alpha}\|^2)$ and Jacobian leakage is $O(\|\boldsymbol{\alpha}\|_\infty)$, controlling the departure from the FP locus. On a $300\mathrm{M}\to857\mathrm{M}$ Transformer adapted from WikiText-103 to BookCorpus, gate-zero growth achieves near-zero old-domain forgetting ($\Delta_A$).
Blogger's Review: Gate-zero growth offers an innovative geometric perspective for continual learning, significantly reducing the forgetting phenomenon of models on new tasks by controlling the interaction between old and new weights. This method not only has theoretical significance but also demonstrates excellent performance in practical applications, making it worthy of further exploration and application.