World models are widely used in offline reinforcement learning (RL) to enhance sample efficiency and generate experiences beyond a fixed dataset. However, they are susceptible to model exploitation in areas with thin data coverage. Previous work typically addresses this by collecting more expert demonstrations, which can be costly, unsafe, or unavailable, or by using conservative algorithms that avoid uncertain regions, limiting generalization.
We propose to directly repair exploitation by leveraging human preferences over imagined rollouts, utilizing the strong intuitive physics that allows humans to easily identify egregious dynamics hallucinations. We formalize this as Dynamics Learning from Human Feedback (DLHF), employing a Bradley-Terry preference loss over trajectory log-likelihoods under a learned dynamics model.
Unfortunately, naive DLHF is sample inefficient, leading us to introduce RENEW, which uses epistemic uncertainty to focus fine-tuning where the model is most exploitable. We evaluate our approach on several Jumanji and classic control environments, finding that while naive DLHF requires an excessive preference budget, RENEW makes the framework practical by improving sample efficiency, limiting catastrophic forgetting, and reducing exploitation in pretrained world models.
Collectively, our results provide initial evidence that preferences can directly supervise world model dynamics, offering a novel approach to addressing exploitation in offline model-based RL.
Blogger's Review: This paper introduces a novel method that integrates human preferences into the optimization of world model dynamics learning. The introduction of RENEW not only enhances sample efficiency but also effectively mitigates model exploitation issues, holding significant theoretical and practical implications. This research paves the way for future applications in offline reinforcement learning.