Catastrophic forgetting remains a fundamental obstacle in supervised continual learning because adapting to new tasks causes representation drift and loss of previously acquired knowledge. The challenge is to constrain the right changes while preserving plasticity. We introduce Sparse Autoencoder Geometric Anchoring (SAGA), which trains a Sparse Autoencoder (SAE) at each task boundary and converts it into local low‑rank constraints on representation drift.
Replay samples define where the constraints apply, and the leading right singular vectors of the encoder Jacobian specify which directions are penalized. The resulting geometry is input‑dependent and can be discarded after snapshotting, without keeping the SAE for later tasks. We show a local equivalence to latent code matching, derive the spectral truncation error, and obtain a conditional bound on frozen‑readout drift.
With replay and snapshot history, SAGA achieves the highest mean Class‑IL accuracy and the lowest mean forgetting on three pretrained ViT‑B/16 benchmarks (CIFAR‑100, CUB‑200, ImageNet‑R), improving over the strongest baseline by 12.15, 4.79, and 3.22 percentage points respectively. Results from training from scratch are more mixed, while rank ablations and geometry controls demonstrate that the SAE‑induced low‑rank geometry provides useful stability‑plasticity trade‑offs.
Review