Abstract
Self-organization is an emergent property of life, driven by the collective behavior of individual components acting on local information. Biological neurons, through local interactions transmitted through synapses, can learn efficiently and adapt their connections over an organism's lifespan.
Motivated by these desirable properties of adaptability and local interaction, neural cellular automata (NCA) models have successfully learned morphogenesis solely through local update rules, demonstrating stability over many updates and robustness to perturbations.
In this work, we introduce Meta Neural Cellular Automata (MetaNCA), a framework that learns local rules which self-organize the weights of artificial neural networks. A learned rule network iteratively updates the weights of a task network using only local interactions on the computation graph. We propose a novel Weight Transformer architecture for the local rule network, which uses linear attention to aggregate signals from neighboring weights and hidden states.
Once trained, the rule network generates task networks of diverse architectures without backpropagation. We show that MetaNCA generates weights for feedforward MLPs, CNNs, and ResNets on MNIST and CIFAR-100, scaling to networks of 2 million parameters.
Furthermore, we demonstrate that MetaNCA generalizes to architectures not seen during meta-training, and that architectural diversity in the training phase strengthens this generalization.
Blogger's Review: The introduction of MetaNCA provides a groundbreaking perspective on self-organizing neural network architectures, achieving diversity and generalization capabilities without reliance on backpropagation. This not only enhances the adaptability of networks but also opens up rich possibilities for future research in neural network design.