Open-ended Multi-agent Autocurricula via Visual Inspection of Policies with Multi-modal LLMs
For multi-agent RL researchers, VIP offers a novel way to assess task difficulty for curriculum learning, achieving stronger performance than existing approaches on a standard benchmark.
The paper introduces Visual Inspection of Policies (VIP), a method that uses a Video Language Model to generate open-ended curricula for multi-agent reinforcement learning by directly inspecting policy behavior via recorded episode videos. On the StarCraft Multi-Agent Challenge, VIP with a lightweight VLM outperforms text-only and scalar-based curriculum methods.
Open-ended curricula in Reinforcement Learning (RL) aim to train generally-capable agents by identifying tasks that facilitate learning increasingly complex skills. A major challenge when designing such curricula is assessing task difficulty relative to the agent's current learning progress. While previous work has explored using scalar task scores or textual summaries of the agent's behavior, here we study a different approach: directly inspecting policy behavior via recorded episode videos. We introduce a simple yet effective instantiation of this approach which leverages a Video Language Model (VLM) to both process these videos and provide curriculum recommendations, which we call Visual Inspection of Policies (VIP). Since videos can naturally contain any number of controllable agents, we empirically study VIP on the StarCraft Multi-Agent Challenge (SMAC). We show that even with a lightweight and openly accessible VLM (VideoLLaMa2-7B), VIP can use policy videos to generate more effective curricula than both its text-only ablation and methods that rely on scalar task scores.