Wonder Wins Ways: Curiosity-Driven Exploration through Multi-Agent Contextual Calibration
This addresses the challenge of efficient exploration in decentralized, communication-free multi-agent systems, which is incremental as it builds on existing curiosity mechanisms by incorporating peer behavior novelty.
The paper tackles the problem of sparse-reward exploration in multi-agent reinforcement learning by proposing CERMIC, a framework that filters noisy curiosity signals and calibrates intrinsic motivation with multi-agent context, resulting in significant performance improvements over state-of-the-art algorithms on benchmarks like VMAS, Meltingpot, and SMACv2.
Autonomous exploration in complex multi-agent reinforcement learning (MARL) with sparse rewards critically depends on providing agents with effective intrinsic motivation. While artificial curiosity offers a powerful self-supervised signal, it often confuses environmental stochasticity with meaningful novelty. Moreover, existing curiosity mechanisms exhibit a uniform novelty bias, treating all unexpected observations equally. However, peer behavior novelty, which encode latent task dynamics, are often overlooked, resulting in suboptimal exploration in decentralized, communication-free MARL settings. To this end, inspired by how human children adaptively calibrate their own exploratory behaviors via observing peers, we propose a novel approach to enhance multi-agent exploration. We introduce CERMIC, a principled framework that empowers agents to robustly filter noisy surprise signals and guide exploration by dynamically calibrating their intrinsic curiosity with inferred multi-agent context. Additionally, CERMIC generates theoretically-grounded intrinsic rewards, encouraging agents to explore state transitions with high information gain. We evaluate CERMIC on benchmark suites including VMAS, Meltingpot, and SMACv2. Empirical results demonstrate that exploration with CERMIC significantly outperforms SoTA algorithms in sparse-reward environments.