Convergence Rates of Posterior Distributions in Markov Decision Process
This work provides foundational theoretical insights for Bayesian reinforcement learning, applicable to both episodic and continuous tasks, though it is incremental in extending existing convergence analyses.
The paper establishes convergence rates for posterior distributions of model dynamics and mean cumulative rewards in Markov Decision Processes (MDPs) and extends these results to Markov games, with theoretical guarantees for general state-action spaces and infinite-dimensional parameters.
In this paper, we show the convergence rates of posterior distributions of the model dynamics in a MDP for both episodic and continuous tasks. The theoretical results hold for general state and action space and the parameter space of the dynamics can be infinite dimensional. Moreover, we show the convergence rates of posterior distributions of the mean accumulative reward under a fixed or the optimal policy and of the regret bound. A variant of Thompson sampling algorithm is proposed which provides both posterior convergence rates for the dynamics and the regret-type bound. Then the previous results are extended to Markov games. Finally, we show numerical results with three simulation scenarios and conclude with discussions.