In the post-training of frontier language models, curated task suites are often used, which inevitably leads to a distribution shift between training and deployment environments. This exposes developers to generalization failures, which are relatively poorly understood. To better understand such failures, we propose that the community should construct clean demonstrations under simplified conditions.
To facilitate this, we introduce a simple and flexible method for constructing language models that fail to generalize in controllable ways when subsequently trained with Reinforcement Learning (RL) on a given distribution of training tasks. Our approach employs Supervised Fine-Tuning on a dataset comprising a mixture of transcripts corresponding to a collection of 'conditional policies', each independently assigned specific behaviors across different task distributions, yielding a model that can be approximated as a 'mixture of conditional policies'.
We observe that RL training selects for policies that yield the highest reward on the training distribution, potentially leading to striking behaviors: in a controlled setting where two distributions contain identical questions prefixed with different 'trigger strings', RL training on either distribution actively degrades performance on the other to zero, even though the underlying task remains identical.
Furthermore, we use our construction to illustrate two novel ways in which generalization may fail in future language models, corresponding to distribution shifts of task coverage and temporal context, respectively. While our construction is intentionally simple and may not closely resemble 'natural' generalization failures, the resulting 'model organisms' are significant for alignment stress-testing and generalization science, serving as existence proofs that training success and generalization can diverge in structured ways.
Blogger's Review: This research delves into the issue of generalization failures in RL training, particularly under distribution shifts, revealing the fragility of models across different tasks. It provides crucial insights for the design and evaluation of future language models, especially in the realms of alignment and generalization studies. By constructing 'model organisms', researchers can better understand complex generalization mechanisms in simpler environments.