Stein Variational Policy Gradient
This work addresses inefficiencies in reinforcement learning for continuous control, though it is incremental as it builds on existing policy gradient methods.
The authors tackled the problems of high variance, slow convergence, and inefficient exploration in policy gradient methods for reinforcement learning by introducing a maximum entropy policy optimization framework and a novel Stein variational policy gradient method (SVPG). They showed that SVPG improves average return and data efficiency when implemented on top of REINFORCE and advantage actor-critic algorithms on continuous control problems.
Policy gradient methods have been successfully applied to many complex reinforcement learning problems. However, policy gradient methods suffer from high variance, slow convergence, and inefficient exploration. In this work, we introduce a maximum entropy policy optimization framework which explicitly encourages parameter exploration, and show that this framework can be reduced to a Bayesian inference problem. We then propose a novel Stein variational policy gradient method (SVPG) which combines existing policy gradient methods and a repulsive functional to generate a set of diverse but well-behaved policies. SVPG is robust to initialization and can easily be implemented in a parallel manner. On continuous control problems, we find that implementing SVPG on top of REINFORCE and advantage actor-critic algorithms improves both average return and data efficiency.