Enabling lifelong learning in large language models (LLMs) requires addressing the stability-plasticity dilemma, where models must incorporate new knowledge without overwriting prior representations while maintaining scalability under bounded parameter growth. Existing parameter-efficient fine-tuning (PEFT) methods fail to satisfy this triad; shared-parameter approaches suffer from catastrophic forgetting, while task-isolated expansions hinder knowledge transfer and scale linearly.
We propose Mixtures of SubExperts (MoSEs), a modular and sparse framework that factorizes model capacity into reusable, compositional primitives. MoSEs enhance transformer layers with lightweight SubExperts and a learned sub-routing function that dynamically selects and composes a sparse subset of modules based on task inputs. This induces a structured decomposition of the parameter space where knowledge is localized yet accessible, mitigating interference while preserving reuse.
Specifically, MoSEs balance the dilemma through three pillars: (i) stability by isolating knowledge within sparsely activated modules, (ii) plasticity through routing-driven recombination and selective expansion, and (iii) scalability via sublinear growth in effective capacity. Notably, the routing mechanism enables compositional generalization, allowing new tasks to be represented as combinations of previously acquired sub-functions. We empirically validate MoSEs on TRACE and SuperNI, showing reduced forgetting, improved forward transfer, and better parameter efficiency over strong PEFT baselines.
MoSEs establish a new Pareto frontier, achieving state-of-the-art performance while maintaining strict parameter budgets. Our results suggest that modular sparsity and compositional routing are key inductive biases for building foundation models that continually learn without saturation.
Blogger's Review: The introduction of MoSEs presents a novel solution to the challenges of lifelong learning in large language models, particularly in balancing knowledge reuse and interference. By leveraging modularity and sparsity, MoSEs enhance model efficiency and achieve new performance heights, showcasing the potential and direction for future advancements in large models.