Scalable and Sample Efficient Distributed Policy Gradient Algorithms in Multi-Agent Networked Systems
This work addresses scalable and efficient learning in multi-agent networked systems, such as wireless networks, with incremental improvements in distributed policy optimization.
The paper tackles the problem of multi-agent reinforcement learning with reward coupling but independent state transitions (REC-MARL), presenting a distributed policy gradient algorithm that achieves stationary policies with iterative complexity bounds based on local dimensions, and experimental results show it significantly outperforms state-of-the-art algorithms and benchmarks in applications like real-time access control and power control in wireless networks.
This paper studies a class of multi-agent reinforcement learning (MARL) problems where the reward that an agent receives depends on the states of other agents, but the next state only depends on the agent's own current state and action. We name it REC-MARL standing for REward-Coupled Multi-Agent Reinforcement Learning. REC-MARL has a range of important applications such as real-time access control and distributed power control in wireless networks. This paper presents a distributed policy gradient algorithm for REC-MARL. The proposed algorithm is distributed in two aspects: (i) the learned policy is a distributed policy that maps a local state of an agent to its local action and (ii) the learning/training is distributed, during which each agent updates its policy based on its own and neighbors' information. The learned algorithm achieves a stationary policy and its iterative complexity bounds depend on the dimension of local states and actions. The experimental results of our algorithm for the real-time access control and power control in wireless networks show that our policy significantly outperforms the state-of-the-art algorithms and well-known benchmarks.