On the Convergence of Discounted Policy Gradient Methods
This addresses a theoretical gap for researchers in reinforcement learning, providing convergence guarantees for widely used but previously unanalyzed discounted approximations.
The paper tackles the convergence behavior of discounted policy gradient methods in reinforcement learning, showing that by increasing the discount factor slowly in relation to a decreasing learning rate, the method achieves standard gradient ascent guarantees on the undiscounted objective.
Many popular policy gradient methods for reinforcement learning follow a biased approximation of the policy gradient known as the discounted approximation. While it has been shown that the discounted approximation of the policy gradient is not the gradient of any objective function, little else is known about its convergence behavior or properties. In this paper, we show that if the discounted approximation is followed such that the discount factor is increased slowly at a rate related to a decreasing learning rate, the resulting method recovers the standard guarantees of gradient ascent on the undiscounted objective.