Variance Reduction in Actor Critic Methods (ACM)
This provides a theoretical justification for deep policy gradient methods, addressing variance issues in reinforcement learning.
The paper tackles variance reduction in Actor Critic Methods by proving that QAC and A2C are optimal control variate estimators, leading to a new formulation with lower variance that improves traditional A2C.
After presenting Actor Critic Methods (ACM), we show ACM are control variate estimators. Using the projection theorem, we prove that the Q and Advantage Actor Critic (A2C) methods are optimal in the sense of the $L^2$ norm for the control variate estimators spanned by functions conditioned by the current state and action. This straightforward application of Pythagoras theorem provides a theoretical justification of the strong performance of QAC and AAC most often referred to as A2C methods in deep policy gradient methods. This enables us to derive a new formulation for Advantage Actor Critic methods that has lower variance and improves the traditional A2C method.