Abstract
The alignment of Small Language Models (SLMs) in the 70--500M parameter range is often considered unstable, yet the underlying failure mechanisms have not been systematically investigated. In the latest research, fifteen (model, corpus) configurations were trained using Proximal Policy Optimization (PPO). The experiments included Pythia-70M, 160M, 410M and SmolLM2-135M, 360M on the TinyStories, CNN/DailyMail, and Wikitext-103 corpora.
Failure Modes
Three reproducible failure modes were identified in small-scale language models:
- Silent LoRA parameter freezing in standard PEFT/TRL pipelines.
- Numerical overflow in importance ratios when using bfloat16.
- Catastrophic policy collapse due to reward-model error.
Solutions
To address these issues, the following solutions were proposed:
- Merge-and-reinitialize adapter technique.
- Using float32 precision during PPO updates.
- A three-layer safety mechanism including reward whitening, importance-ratio guarding, and weight rollback.
Hypothesis
This paper proposes a capacity-headroom hypothesis, suggesting that PPO performance at the SLM scale depends on a fluent supervised model ($\text{PPL}$). These insights offer new avenues for enhancing the stability and performance of small-scale language models.
Blogger's Review: This paper delves into the stability issues of small-scale language models in reinforcement learning, identifying key failure modes and providing actionable solutions. The findings are crucial for future research and applications, particularly on how to effectively leverage reinforcement learning techniques in resource-constrained environments.