Mean Actor Critic
This addresses variance reduction in policy gradients for reinforcement learning practitioners, but it is incremental as it builds on existing actor-critic methods.
The paper tackles the problem of high variance in policy gradient estimates for discrete-action continuous-state reinforcement learning by proposing Mean Actor-Critic (MAC), which uses all action values to estimate gradients, and shows it is competitive with state-of-the-art algorithms on control domains and Atari games.
We propose a new algorithm, Mean Actor-Critic (MAC), for discrete-action continuous-state reinforcement learning. MAC is a policy gradient algorithm that uses the agent's explicit representation of all action values to estimate the gradient of the policy, rather than using only the actions that were actually executed. We prove that this approach reduces variance in the policy gradient estimate relative to traditional actor-critic methods. We show empirical results on two control domains and on six Atari games, where MAC is competitive with state-of-the-art policy search algorithms.