Analysis of On-policy Policy Gradient Methods under the Distribution Mismatch
This work addresses a theoretical gap between policy gradient theory and practical implementations, which is important for researchers in reinforcement learning, though it is incremental in nature.
The paper analyzes the impact of distribution mismatch on policy gradient methods in reinforcement learning, showing that they remain globally optimal in tabular cases and extending this to general parameterizations using biased stochastic gradient descent theory.
Policy gradient methods are one of the most successful methods for solving challenging reinforcement learning problems. However, despite their empirical successes, many SOTA policy gradient algorithms for discounted problems deviate from the theoretical policy gradient theorem due to the existence of a distribution mismatch. In this work, we analyze the impact of this mismatch on the policy gradient methods. Specifically, we first show that in the case of tabular parameterizations, the methods under the mismatch remain globally optimal. Then, we extend this analysis to more general parameterizations by leveraging the theory of biased stochastic gradient descent. Our findings offer new insights into the robustness of policy gradient methods as well as the gap between theoretical foundations and practical implementations.