In model-based reinforcement-learning agents of the DreamerV3 family, catastrophic forgetting occurs when trained on task sequences, even with an unbounded replay buffer preserving all prior experiences. We pose a question that the continual RL literature has assumed but never measured: which component forgets? Under unclear replay strategies, pre-registered component-level probes (n=3 seeds) reveal that the world model retains nearly all measurable information about old tasks — reward discrimination (retention ratio ~1.0), value estimates, and termination structure — while the actor's performance collapses. Forgetting in this regime is a channel problem, not a memory problem. We demonstrate this through intervention: with the world model frozen and identical imagined rollouts, reinforcement learning in imagination fails to recover a lost skill (0/3 seeds), while supervised self-imitation on the world model's own graded dreams successfully recovers it in 3/3 seeds with zero environment interaction.
Interleaved during training, this graded dream rehearsal yields a task-label-free, parameter-constant continual learner: 3/3 four-task chains retained where plain replay passes 0/3, 3/3 eight-task chains, and consistent gains over matched real-episode cloning (paired difference +0.13, bootstrap 95% CI [0.07, 0.24], complete seed separation). The dream-grading step is load-bearing: we characterize two scoring failure modes, provide an offline selection gauge that caught both before they contaminated results, and give a realized-first grading rule that closes them. All experiments were pre-registered with committed protocols; every refuted hypothesis is reported.
Blogger's Review: This paper reveals the nuanced relationship between the stability of the world model and the forgetting of the actor in continual reinforcement learning, emphasizing the significance of dream rehearsal in skill recovery. This opens new perspectives for future research, particularly in the potential applications of task transfer and continual learning, warranting deeper exploration.