Derivative-Free Policy Optimization for Linear Risk-Sensitive and Robust Control Design: Implicit Regularization and Sample Complexity
This work provides theoretical guarantees for derivative-free policy optimization methods in risk-sensitive and robust control, which is important for safety-critical control systems. It also offers the first sample complexity for PG methods in zero-sum linear-quadratic dynamic games, an incremental step for multi-agent reinforcement learning.
This paper investigates policy gradient (PG) methods for learning linear risk-sensitive and robust controllers, establishing global convergence and sample complexity for finite-horizon linear exponential quadratic Gaussian and linear-quadratic disturbance attenuation problems. As a result, it also provides the first sample complexity for global convergence of PG methods in zero-sum linear-quadratic dynamic games.
Direct policy search serves as one of the workhorses in modern reinforcement learning (RL), and its applications in continuous control tasks have recently attracted increasing attention. In this work, we investigate the convergence theory of policy gradient (PG) methods for learning the linear risk-sensitive and robust controller. In particular, we develop PG methods that can be implemented in a derivative-free fashion by sampling system trajectories, and establish both global convergence and sample complexity results in the solutions of two fundamental settings in risk-sensitive and robust control: the finite-horizon linear exponential quadratic Gaussian, and the finite-horizon linear-quadratic disturbance attenuation problems. As a by-product, our results also provide the first sample complexity for the global convergence of PG methods on solving zero-sum linear-quadratic dynamic games, a nonconvex-nonconcave minimax optimization problem that serves as a baseline setting in multi-agent reinforcement learning (MARL) with continuous spaces. One feature of our algorithms is that during the learning phase, a certain level of robustness/risk-sensitivity of the controller is preserved, which we termed as the implicit regularization property, and is an essential requirement in safety-critical control systems.