On the Gradient Domination of the LQG Problem
This work addresses a theoretical gap in reinforcement learning for control systems, providing convergence guarantees for LQG problems, which is incremental as it extends known results from LQR to a more complex setting.
The paper tackles the lack of gradient dominance in policy gradient methods for the linear quadratic Gaussian (LQG) problem by introducing a history-based parameterization, establishing gradient dominance and approximate smoothness to prove global convergence and per-iteration stability guarantees in model-based and model-free settings, with numerical experiments on an unstable system supporting these results.
We consider solutions to the linear quadratic Gaussian (LQG) regulator problem via policy gradient (PG) methods. Although PG methods have demonstrated strong theoretical guarantees in solving the linear quadratic regulator (LQR) problem, despite its nonconvex landscape, their theoretical understanding in the LQG setting remains limited. Notably, the LQG problem lacks gradient dominance in the classical parameterization, i.e., with a dynamic controller, which hinders global convergence guarantees. In this work, we study PG for the LQG problem by adopting an alternative parameterization of the set of stabilizing controllers and employing a lifting argument. We refer to this parameterization as a history representation of the control input as it is parameterized by past input and output data from the previous p time-steps. This representation enables us to establish gradient dominance and approximate smoothness for the LQG cost. We prove global convergence and per-iteration stability guarantees for policy gradient LQG in model-based and model-free settings. Numerical experiments on an open-loop unstable system are provided to support the global convergence guarantees and to illustrate convergence under different history lengths of the history representation.