We address the problem of safely training an agent policy in settings where environment dynamics are unknown and no suitable reward function is available. Given the impracticality of traditional reinforcement learning in safety-critical environments, we turn to human input as a resource. We introduce DROPJ, a human-centered method for safe training and deployment.
First, we learn a world model (a learned simulator) from a dataset of prior real-world trajectories. A human then plays the game in this learned simulator to extract several informative simulated trajectories.
We sample pairs of simulated trajectory segments from these and elicit human preferences over these segments, along with justifications for their choices.
Using these justified preferences, we train a reward model and deploy the agent using model predictive control in conjunction with the world model.
Our real-user experiments show that generating informative simulated trajectories from users significantly reduces computational costs during training and improves performance during deployment.
We demonstrate that using preferences rather than other feedback types substantially enhances deployment performance. Additionally, safety justifications accompanying preferences significantly enhance safety and prioritize user-defined safety aspects during deployment.
Blogger's Review: This research presents an innovative approach to training and deploying safe agents by integrating human preferences and justifications, highlighting the critical role of human knowledge in complex environments. It signifies a pivotal advancement in the realm of reinforcement learning.