Agent-Time Attention for Sparse Rewards Multi-Agent Reinforcement Learning
This work addresses the problem of credit assignment in collaborative MARL for environments with sparse rewards, which is incremental as it builds on existing MARL methods by introducing a novel attention mechanism.
The paper tackles the challenge of sparse and delayed rewards in multi-agent reinforcement learning (MARL) by proposing Agent-Time Attention (ATA), a neural network model with auxiliary losses for reward redistribution, and demonstrates that ATA outperforms various baselines on extended MiniGrid environments like MultiRoom and DoorKey.
Sparse and delayed rewards pose a challenge to single agent reinforcement learning. This challenge is amplified in multi-agent reinforcement learning (MARL) where credit assignment of these rewards needs to happen not only across time, but also across agents. We propose Agent-Time Attention (ATA), a neural network model with auxiliary losses for redistributing sparse and delayed rewards in collaborative MARL. We provide a simple example that demonstrates how providing agents with their own local redistributed rewards and shared global redistributed rewards motivate different policies. We extend several MiniGrid environments, specifically MultiRoom and DoorKey, to the multi-agent sparse delayed rewards setting. We demonstrate that ATA outperforms various baselines on many instances of these environments. Source code of the experiments is available at https://github.com/jshe/agent-time-attention.