In the field of reinforcement learning (RL), it is commonly observed that online RL outperforms offline methods across various performance metrics. Specifically, RL-trained policies exhibit superior out-of-distribution (OOD) behavior, while models relying solely on imitation learning often struggle. A recent study introduced an OOD-focused benchmark, reporting that RL-trained vision-language-action (VLA) policies achieve significantly better OOD performance than their supervised fine-tuning (SFT) counterparts, with slight improvements in in-distribution (IND) performance as well.
This work investigates whether hybrid offline-online training can leverage the advantages of both approaches. Specifically, we explore RL methods regularized by offline supervision through either offline data or an offline-trained reference policy. We evaluate these approaches on the OOD benchmark and compare them against both offline-only training and standard RL.
Our results indicate that while offline training achieves limited OOD performance on its own, incorporating offline supervision into RL preserves strong OOD capabilities while significantly enhancing training efficiency. Notably, the guided methods reach performance levels close to those of standard RL while requiring approximately half the training budget. Rather than creating a trade-off between speed and OOD performance, the hybrid approach retains robust OOD capability while achieving this efficiency gain.
Blogger's Review: This paper presents an efficient training method by integrating offline supervision with reinforcement learning, addressing the limitations of traditional RL in OOD performance. The hybrid approach not only maintains strong OOD capabilities but also significantly reduces training costs, offering new insights for future development of vision-language-action models.