Planning with Information-Processing Constraints and Model Uncertainty in Markov Decision Processes
This work addresses planning challenges in MDPs for AI and control systems by integrating model uncertainty with information constraints, though it appears incremental as it builds on existing variational principles.
The authors tackled the problem of planning in Markov Decision Processes (MDPs) under both information-processing constraints and model uncertainty by deriving a unified solution from a generalized variational principle, resulting in a value iteration scheme that includes standard, Bayesian, and robust planning as special cases and demonstrating benefits in a grid world simulation.
Information-theoretic principles for learning and acting have been proposed to solve particular classes of Markov Decision Problems. Mathematically, such approaches are governed by a variational free energy principle and allow solving MDP planning problems with information-processing constraints expressed in terms of a Kullback-Leibler divergence with respect to a reference distribution. Here we consider a generalization of such MDP planners by taking model uncertainty into account. As model uncertainty can also be formalized as an information-processing constraint, we can derive a unified solution from a single generalized variational principle. We provide a generalized value iteration scheme together with a convergence proof. As limit cases, this generalized scheme includes standard value iteration with a known model, Bayesian MDP planning, and robust planning. We demonstrate the benefits of this approach in a grid world simulation.