As autonomous agents become increasingly deployed across diverse operational contexts, aligning their behavior with human intent poses a significant challenge. This necessitates reward functions that are robust to environmental changes rather than overfitting to a single context. Inverse reinforcement learning (IRL) offers a principled way to infer these objectives from human feedback. However, existing analyses of optimal teaching methods for IRL primarily focus on single-environment, demonstration-only settings, leaving underexplored how heterogeneous feedback modalities and environment dynamics jointly constrain reward functions that generalize across multiple environments.
We first analyze how different feedback modalities constrain rewards, showing that, in the unlimited-data regime, comparisons impose strictly stronger global constraints than other modalities.
Beyond this theoretical analysis, we introduce a hierarchical machine teaching algorithm for reward learning that operates across multiple MDPs. The algorithm first greedily selects informative environments that expose complementary reward constraints, then strategically queries low-cost feedback within those environments.
Empirically, our method achieves substantially lower regret and stronger generalization to held-out environments than uniform teaching baselines under identical feedback budgets, demonstrating the importance of multi-environment, multi-modal teaching for learning dynamics-robust reward functions.
Blogger's Review: This paper provides an innovative machine teaching algorithm through a deep analysis of multi-environment and multi-modal feedback, showcasing its potential in reward learning. It presents new insights for future applications of autonomous agents in complex environments, especially where adherence to human intent is crucial. The empirical results highlight the significance of effective feedback selection strategies, warranting further exploration and application.