Open-ended curricula in Reinforcement Learning (RL) aim to train generally-capable agents by identifying tasks that facilitate learning increasingly complex skills. A major challenge when designing such curricula is assessing task difficulty relative to the agent's current learning progress. While previous work has explored using scalar task scores or textual summaries of the agent's behavior, here we study a different approach: directly inspecting policy behavior via recorded episode videos. We introduce a simple yet effective instantiation of this approach which leverages a Video Language Model (VLM) to process these videos and provide curriculum recommendations, called Visual Inspection of Policies (VIP). Since videos can contain any number of controllable agents, we empirically study VIP on the StarCraft Multi-Agent Challenge (SMAC). We show that even with a lightweight and openly accessible VLM (VideoLLaMa2-7B), VIP can generate more effective curricula than both its text-only ablation and methods that rely on scalar task scores.
Blogger's Review: This study introduces an innovative approach to optimize multi-agent learning curricula through video analysis, showcasing the potential of video language models in Reinforcement Learning. This method may enhance agent learning capabilities and provide new directions for future research.