Continual learning (CL) aims to adapt pretrained models to new tasks while preserving their existing capabilities. The preservation tasks that are supplied, however, only cover a fraction of what a model knows, and forgetting can extend beyond them. Recent work shows that language models can generate useful replay data without accessing the original training corpus, thereby supplementing the limited evidence. This raises the question of self‑preservation: can a model help choose preservation targets that protect its capabilities more broadly?
To investigate, we organize task descriptions and their examples into Task Banks. Entries from these banks can be selected and combined to guide replay generation by a frozen model. We perform continual instruction tuning on Qwen models of various sizes and observe three key findings.
- The new task and the learning rate shape which prior capabilities degrade; tracking previously correct answers reveals losses that average accuracy can hide.
- Generated replay reduces forgetting on tasks outside the bank; merging partial banks strengthens this protection, yielding higher mean retention than using either bank alone.
- The model can also assist in selecting replay targets: at the 4B scale, selection informed by the new task improves mean held‑out retention over random selection and provides comparable new‑task gains, while smaller models show no clear advantage. Under a fixed example budget, replacing new‑task examples with replay improves preservation at the cost of smaller new‑task gains.
Together, these results demonstrate that language models can support broader self‑preservation by generating replay data and, in certain settings, by choosing which tasks to replay. Review