Exploiting Semantic Epsilon Greedy Exploration Strategy in Multi-Agent Reinforcement Learning
This addresses exploration inefficiencies in MARL for applications like gaming or robotics, but it is incremental as it builds on existing QMIX with a novel exploration strategy.
The paper tackles the problem of inefficient exploration in multi-agent reinforcement learning (MARL) by proposing QMIX(SEG), which integrates a Semantic Epsilon Greedy (SEG) strategy to cluster actions into groups for bi-level exploration, resulting in performance that largely outperforms QMIX and is competitive with state-of-the-art methods on the SMAC benchmark.
Multi-agent reinforcement learning (MARL) can model many real world applications. However, many MARL approaches rely on epsilon greedy for exploration, which may discourage visiting advantageous states in hard scenarios. In this paper, we propose a new approach QMIX(SEG) for tackling MARL. It makes use of the value function factorization method QMIX to train per-agent policies and a novel Semantic Epsilon Greedy (SEG) exploration strategy. SEG is a simple extension to the conventional epsilon greedy exploration strategy, yet it is experimentally shown to greatly improve the performance of MARL. We first cluster actions into groups of actions with similar effects and then use the groups in a bi-level epsilon greedy exploration hierarchy for action selection. We argue that SEG facilitates semantic exploration by exploring in the space of groups of actions, which have richer semantic meanings than atomic actions. Experiments show that QMIX(SEG) largely outperforms QMIX and leads to strong performance competitive with current state-of-the-art MARL approaches on the StarCraft Multi-Agent Challenge (SMAC) benchmark.