Natural Actor-Critic Converges Globally for Hierarchical Linear Quadratic Regulator
This work addresses a fundamental theoretical gap in multi-agent reinforcement learning for researchers, providing a scalable solution with proven convergence, though it is incremental as it builds on existing LQR frameworks.
The paper tackles the curse of dimensionality in multi-agent reinforcement learning by studying a hierarchical linear quadratic regulator (LQR) problem with partially exchangeable agents, developing a hierarchical actor-critic algorithm with computational complexity independent of the total number of agents and proving its global linear convergence to the optimal policy.
Multi-agent reinforcement learning has been successfully applied to a number of challenging problems. Despite these empirical successes, theoretical understanding of different algorithms is lacking, primarily due to the curse of dimensionality caused by the exponential growth of the state-action space with the number of agents. We study a fundamental problem of multi-agent linear quadratic regulator (LQR) in a setting where the agents are partially exchangeable. In this setting, we develop a hierarchical actor-critic algorithm, whose computational complexity is independent of the total number of agents, and prove its global linear convergence to the optimal policy. As LQRs are often used to approximate general dynamic systems, this paper provides an important step towards a better understanding of general hierarchical mean-field multi-agent reinforcement learning.