ColorGrid: A Multi-Agent Non-Stationary Environment for Goal Inference and Assistance
This work addresses the need for better benchmarks in multi-agent reinforcement learning for human-robot collaboration, though it is incremental as it focuses on environment design rather than algorithmic breakthroughs.
The authors tackled the problem of evaluating autonomous agents' ability to infer human goals in multi-agent settings by introducing ColorGrid, a novel MARL environment with customizable non-stationarity and asymmetry, and found that a state-of-the-art algorithm (IPPO) failed to solve it under specific conditions.
Autonomous agents' interactions with humans are increasingly focused on adapting to their changing preferences in order to improve assistance in real-world tasks. Effective agents must learn to accurately infer human goals, which are often hidden, to collaborate well. However, existing Multi-Agent Reinforcement Learning (MARL) environments lack the necessary attributes required to rigorously evaluate these agents' learning capabilities. To this end, we introduce ColorGrid, a novel MARL environment with customizable non-stationarity, asymmetry, and reward structure. We investigate the performance of Independent Proximal Policy Optimization (IPPO), a state-of-the-art (SOTA) MARL algorithm, in ColorGrid and find through extensive ablations that, particularly with simultaneous non-stationary and asymmetric goals between a ``leader'' agent representing a human and a ``follower'' assistant agent, ColorGrid is unsolved by IPPO. To support benchmarking future MARL algorithms, we release our environment code, model checkpoints, and trajectory visualizations at https://github.com/andreyrisukhin/ColorGrid.