Abstract
Reinforcement Learning is commonly used to train large language models using environmental feedback. However, in applied settings, the environment usually provides sparse or delayed feedback, making it challenging for the model to pinpoint which actions in its reasoning led to success or failure. Thus, learning effectively from these signals is hard, as the model must determine how each failure should inform meaningful behavioral corrections in subsequent iterations.
We introduce a training framework called Self-Review Reinforcement Learning (SRRL), which embeds an explicit self-review step into each RL episode. When a first-pass response fails, the model generates a self-review to identify what went wrong, conditioning an improved second attempt. Unlike inference-time reflection approaches, such as Reflexion, this framework optimizes self-review with policy gradients and internalizes improvements into the base policy via selective distillation, ensuring they persist across future episodes.
A cross-episode memory keeps successful self-reviews for reuse when encountering similar tasks in future episodes during training. We evaluate SRRL against a standard RLVR baseline using the GRPO optimizer across two language models, Qwen 3-4B and OLMo-3-7B, on the GSM8K benchmark. SRRL consistently outperforms RLVR in final reward performance and achieves greater learning efficiency by successfully transforming feedback into behavioral improvement.
Blogger's Review: The SRRL framework enhances the efficiency of reinforcement learning by introducing a self-review mechanism. This approach not only boosts the model's learning capabilities but also provides valuable memory reserves for future tasks, making it applicable in a wide range of scenarios.