In a continual learning setting, a model must be plastic enough to learn new tasks while remaining stable enough to not disturb previously learned capabilities. This dilemma is argued to have an architectural root. A finite network has limited representational and plastic resources, and the required capacity depends on unknown properties of future task streams: how many tasks will be encountered, and how much they overlap in feature space. Regularization-based methods preserve past knowledge within fixed-capacity architectures, implicitly relying on an oracle architecture sized for this unknown future.
When tasks are weakly related, fixed architectures progressively run out of plastic resources; when tasks are few or strongly overlapping, models are often over-provisioned. Inspired by neurogenesis in biology, we propose NORACL to address the stability-plasticity dilemma by tackling the oracle architecture problem through neuronal growth. Starting from a compact network, NORACL grows only when needed by monitoring two complementary signals for representational and plasticity saturation.
We evaluate NORACL against oracle-sized static baselines across varying task counts and geometries. Across all settings, NORACL achieves final average accuracies that are better than or on par with oracle-provisioned static baselines while using fewer parameters. Additionally, NORACL yields architectures with interpretable growth, where dissimilar tasks predominantly expand feature-extraction layers, while tasks relying on common features shift growth toward later feature-combination layers. Our analysis further explains why fixed-capacity networks lose plasticity as tasks accumulate, whereas NORACL creates fresh capacity for new tasks through growth. Together, these results show that adaptive neurogenesis pushes the stability-plasticity Pareto frontier of continual learning.
Blogger's Review: The introduction of NORACL offers an innovative approach to tackling the stability-plasticity dilemma in continual learning, simulating biological neurogenesis mechanisms. This method not only enhances model performance but also provides new directions for future research, highlighting its potential for effectively addressing diverse tasks under resource constraints.