Sample-Efficient Model-Free Policy Gradient Methods for Stochastic LQR via Robust Linear Regression
Provides convergence guarantees for Natural Policy Gradient and Gauss-Newton methods in unknown stochastic LQR, addressing the errors-in-variables issue for practitioners in control and reinforcement learning.
The paper addresses the challenge of unbiased gradient estimation in policy gradient methods for stochastic LQR, achieving a sample complexity of O(1/epsilon) using a primal-dual estimation procedure.
Policy gradient algorithms are widely used in reinforcement learning and belong to the class of approximate dynamic programming methods. This paper studies two key policy gradient algorithms, the Natural Policy Gradient and the Gauss-Newton Method, for solving the Linear Quadratic Regulator (LQR) problem in unknown stochastic linear systems. The main challenge lies in obtaining an unbiased gradient estimate from noisy data due to errors-in-variables in linear regression. This issue is addressed by employing a primal-dual estimation procedure. Using this novel gradient estimation scheme, the paper establishes convergence guarantees with a sample complexity of order O(1/epsilon). Theoretical results are further supported by numerical experiments, which demonstrate the effectiveness of the proposed algorithms.