Abstract
Offline Reinforcement Learning (ORL) promises safe and sample-efficient deployment, but existing methods rely on undocumented online interactions for hyperparameter tuning and lack reliable fully offline estimates of initial online performance. We introduce SOReL, a fully offline Bayesian model-based RL method that learns a posterior over dynamics, estimates policy value via predictive uncertainty, and enables complete offline hyperparameter selection.
We further propose TOReL, which extends this tuning framework to arbitrary model-free and model-based ORL algorithms. We provide a regret analysis showing that Bayesian offline RL achieves the minimax-optimal parametric rate under standard regularity conditions. Together, our methods establish a practical and theoretically grounded framework for fully offline RL.
Blogger's Review: The SOReL and TOReL methods presented in this paper mark a significant advancement in the offline reinforcement learning domain, particularly in hyperparameter selection and policy value estimation. The introduction of Bayesian approaches enhances the safety and efficiency of offline learning, paving the way for future research in this area.