Exploiting inter-agent coupling information for efficient reinforcement learning of cooperative LQR
This addresses the challenge of efficient multi-agent reinforcement learning for control tasks, though it appears incremental as it builds on existing decomposition approaches.
The paper tackles the problem of developing scalable reinforcement learning for cooperative multi-agent control by proposing a method to exactly decompose local Q-functions using inter-agent coupling information, resulting in improved sample and computational efficiency demonstrated in numerical examples.
Developing scalable and efficient reinforcement learning algorithms for cooperative multi-agent control has received significant attention over the past years. Existing literature has proposed inexact decompositions of local Q-functions based on empirical information structures between the agents. In this paper, we exploit inter-agent coupling information and propose a systematic approach to exactly decompose the local Q-function of each agent. We develop an approximate least square policy iteration algorithm based on the proposed decomposition and identify two architectures to learn the local Q-function for each agent. We establish that the worst-case sample complexity of the decomposition is equal to the centralized case and derive necessary and sufficient graphical conditions on the inter-agent couplings to achieve better sample efficiency. We demonstrate the improved sample efficiency and computational efficiency on numerical examples.