Incremental Model-based Learners With Formal Learning-Time Guarantees
This work addresses scalability issues for researchers and practitioners using model-based reinforcement learning in large-scale problems, but it is incremental as it builds on existing algorithms.
The paper tackles the high computational cost of model-based learning algorithms like RMAX and MBIE in Markov Decision Processes by proposing RTDP-RMAX and RTDP-IE, which are computationally much faster with little loss compared to existing bounds, as shown through theoretical PAC guarantees and experimental evaluation.
Model-based learning algorithms have been shown to use experience efficiently when learning to solve Markov Decision Processes (MDPs) with finite state and action spaces. However, their high computational cost due to repeatedly solving an internal model inhibits their use in large-scale problems. We propose a method based on real-time dynamic programming (RTDP) to speed up two model-based algorithms, RMAX and MBIE (model-based interval estimation), resulting in computationally much faster algorithms with little loss compared to existing bounds. Specifically, our two new learning algorithms, RTDP-RMAX and RTDP-IE, have considerably smaller computational demands than RMAX and MBIE. We develop a general theoretical framework that allows us to prove that both are efficient learners in a PAC (probably approximately correct) sense. We also present an experimental evaluation of these new algorithms that helps quantify the tradeoff between computational and experience demands.