Bayesian Policy Gradients via Alpha Divergence Dropout Inference
This work addresses stability issues in policy gradient methods for continuous control, but it appears incremental as it builds on existing Bayesian and dropout techniques.
The authors tackled the difficulty of deterministic value estimation in continuous control tasks by proposing a Bayesian Neural Network approach that estimates a value function distribution using an α-divergence objective with dropout approximation. They demonstrated improved stability and performance in MuJoCo simulations, though no concrete numbers were provided.
Policy gradient methods have had great success in solving continuous control tasks, yet the stochastic nature of such problems makes deterministic value estimation difficult. We propose an approach which instead estimates a distribution by fitting the value function with a Bayesian Neural Network. We optimize an $α$-divergence objective with Bayesian dropout approximation to learn and estimate this distribution. We show that using the Monte Carlo posterior mean of the Bayesian value function distribution, rather than a deterministic network, improves stability and performance of policy gradient methods in continuous control MuJoCo simulations.