SCC-rFMQ Learning in Cooperative Markov Games with Continuous Actions
This addresses a gap in multiagent reinforcement learning for continuous-action domains, but it appears incremental as it builds on existing methods with a hierarchical approach.
The paper tackles the problem of multiagent coordination in cooperative Markov games with continuous actions, which has been understudied, by proposing SCC-rFMQ, a hierarchical method that samples actions and updates policies. Experimental results on two games show that SCC-rFMQ outperforms other reinforcement learning algorithms.
Although many reinforcement learning methods have been proposed for learning the optimal solutions in single-agent continuous-action domains, multiagent coordination domains with continuous actions have received relatively few investigations. In this paper, we propose an independent learner hierarchical method, named Sample Continuous Coordination with recursive Frequency Maximum Q-Value (SCC-rFMQ), which divides the cooperative problem with continuous actions into two layers. The first layer samples a finite set of actions from the continuous action spaces by a re-sampling mechanism with variable exploratory rates, and the second layer evaluates the actions in the sampled action set and updates the policy using a reinforcement learning cooperative method. By constructing cooperative mechanisms at both levels, SCC-rFMQ can handle cooperative problems in continuous action cooperative Markov games effectively. The effectiveness of SCC-rFMQ is experimentally demonstrated on two well-designed games, i.e., a continuous version of the climbing game and a cooperative version of the boat problem. Experimental results show that SCC-rFMQ outperforms other reinforcement learning algorithms.