Reward-estimation variance elimination in sequential decision processes
This work addresses a key bottleneck in reinforcement learning by improving sample efficiency for researchers and practitioners, though it is incremental as it builds on existing variance-reduction approaches.
The authors tackled the problem of high variance in policy gradient methods for reinforcement learning, particularly in delayed-reward scenarios, and proposed an unbiased method that completely eliminates variance under certain conditions, such as deterministic dynamics and small policy stochasticity, with demonstrated performance in linear quadratic Gaussian models.
Policy gradient methods are very attractive in reinforcement learning due to their model-free nature and convergence guarantees. These methods, however, suffer from high variance in gradient estimation, resulting in poor sample efficiency. To mitigate this issue, a number of variance-reduction approaches have been proposed. Unfortunately, in the challenging problems with delayed rewards, these approaches either bring a relatively modest improvement or do reduce variance at expense of introducing a bias and undermining convergence. The unbiased methods of gradient estimation, in general, only partially reduce variance, without eliminating it completely even in the limit of exact knowledge of the value functions and problem dynamics, as one might have wished. In this work we propose an unbiased method that does completely eliminate variance under some, commonly encountered, conditions. Of practical interest is the limit of deterministic dynamics and small policy stochasticity. In the case of a quadratic value function, as in linear quadratic Gaussian models, the policy randomness need not be small. We use such a model to analyze performance of the proposed variance-elimination approach and compare it with standard variance-reduction methods. The core idea behind the approach is to use control variates at all future times down the trajectory. We present both a model-based and model-free formulations.