Abstract
In RLHF (Reinforcement Learning from Human Feedback), human preferences are typically modeled as a function of the human's reward function or corresponding optimal state-action values. This work proposes that human beliefs about the capabilities of the agent being trained also play a key role in preference generation.
We examine two questions related to this hypothesis, one descriptive and one normative: Do human labelers' beliefs about agent capabilities affect the preferences that they provide? And what is the ideal set of beliefs about an agent -- and resulting preferences -- for humans to have?
We propose a new preference model that incorporates human beliefs and provide a normative theory that bounds the error on the final learned policy based on the mismatch between the human's beliefs and an idealized set of beliefs.
We then confirm via a human study that beliefs about agent capabilities significantly affect preferences and can be influenced through simple interventions. Additionally, we empirically show through synthetic experiments that it is often suboptimal for human preference labelers to assume agent optimality.
Collectively, these results theoretically and empirically demonstrate how reducing the mismatch between human beliefs and agent capabilities can lead to more performant RLHF and point toward new best practices for RLHF practitioners.
Blogger's Review: This paper delves into the role of human beliefs in reinforcement learning, emphasizing the connection between beliefs and preferences. The empirical validation of theoretical hypotheses offers new insights for optimizing RLHF, particularly the finding that beliefs about agent capabilities influence preferences, which provides a significant direction for future research.