Bayesian Policy Optimization for Model Uncertainty
This work addresses robust adaptation to real-world uncertainty for autonomous systems, representing an incremental improvement by building on recent policy optimization algorithms.
The paper tackles the problem of model uncertainty in autonomous systems by formulating it as a continuous Bayes-Adaptive Markov Decision Process (BAMDP) and proposes Bayesian Policy Optimization, a method that learns a universal policy to maximize the Bayesian value function, significantly outperforming algorithms that do not explicitly reason about belief distributions and being competitive with state-of-the-art Partially Observable Markov Decision Process solvers.
Addressing uncertainty is critical for autonomous systems to robustly adapt to the real world. We formulate the problem of model uncertainty as a continuous Bayes-Adaptive Markov Decision Process (BAMDP), where an agent maintains a posterior distribution over latent model parameters given a history of observations and maximizes its expected long-term reward with respect to this belief distribution. Our algorithm, Bayesian Policy Optimization, builds on recent policy optimization algorithms to learn a universal policy that navigates the exploration-exploitation trade-off to maximize the Bayesian value function. To address challenges from discretizing the continuous latent parameter space, we propose a new policy network architecture that encodes the belief distribution independently from the observable state. Our method significantly outperforms algorithms that address model uncertainty without explicitly reasoning about belief distributions and is competitive with state-of-the-art Partially Observable Markov Decision Process solvers.