Inferring the Optimal Policy using Markov Chain Monte Carlo
This addresses the issue of unstable convergence and lack of optimality guarantees in policy gradient methods for applications like video games and robot control.
The paper tackles the problem of estimating the optimal stochastic control policy in model-free reinforcement learning with unknown dynamics and rewards, proposing a Markov Chain Monte Carlo method that provably converges to the globally optimal policy and shows empirically similar variance to policy gradient.
This paper investigates methods for estimating the optimal stochastic control policy for a Markov Decision Process with unknown transition dynamics and an unknown reward function. This form of model-free reinforcement learning comprises many real world systems such as playing video games, simulated control tasks, and real robot locomotion. Existing methods for estimating the optimal stochastic control policy rely on high variance estimates of the policy descent. However, these methods are not guaranteed to find the optimal stochastic policy, and the high variance gradient estimates make convergence unstable. In order to resolve these problems, we propose a technique using Markov Chain Monte Carlo to generate samples from the posterior distribution of the parameters conditioned on being optimal. Our method provably converges to the globally optimal stochastic policy, and empirically similar variance compared to the policy gradient.