Analyzing the Variance of Policy Gradient Estimators for the Linear-Quadratic Regulator
This work provides incremental insights into variance analysis for policy gradient methods, targeting researchers in reinforcement learning.
The paper analyzed the variance of the REINFORCE policy gradient estimator in linear-quadratic regulator environments with continuous spaces and Gaussian noise, deriving theoretical bounds and validating them through simulations.
We study the variance of the REINFORCE policy gradient estimator in environments with continuous state and action spaces, linear dynamics, quadratic cost, and Gaussian noise. These simple environments allow us to derive bounds on the estimator variance in terms of the environment and noise parameters. We compare the predictions of our bounds to the empirical variance in simulation experiments.