Monte Carlo Bayesian Reinforcement Learning
This provides a general and simpler approach to Bayesian reinforcement learning, addressing uncertainty in model parameters for researchers and practitioners in AI.
The paper tackles the problem of Bayesian reinforcement learning by proposing Monte Carlo BRL (MC-BRL), a method that samples model parameters to form a discrete POMDP, resulting in guaranteed performance approximations that handle both fully and partially observable worlds.
Bayesian reinforcement learning (BRL) encodes prior knowledge of the world in a model and represents uncertainty in model parameters by maintaining a probability distribution over them. This paper presents Monte Carlo BRL (MC-BRL), a simple and general approach to BRL. MC-BRL samples a priori a finite set of hypotheses for the model parameter values and forms a discrete partially observable Markov decision process (POMDP) whose state space is a cross product of the state space for the reinforcement learning task and the sampled model parameter space. The POMDP does not require conjugate distributions for belief representation, as earlier works do, and can be solved relatively easily with point-based approximation algorithms. MC-BRL naturally handles both fully and partially observable worlds. Theoretical and experimental results show that the discrete POMDP approximates the underlying BRL task well with guaranteed performance.