Leveraging the Value of Information in POMDP Planning
This work addresses the problem of computational efficiency in POMDP planning for AI and robotics applications, representing an incremental improvement with a novel method for a known bottleneck.
The paper tackles the challenge of efficient planning in partially observable Markov decision processes (POMDPs) under limited time by introducing a framework that exploits varying value of information (VOI) to conditionally process observations, resulting in a Monte Carlo Tree Search algorithm (VOIMCP) that outperforms baselines on benchmarks.
Partially observable Markov decision processes (POMDPs) offer a principled formalism for planning under state and transition uncertainty. Despite advances made towards solving large POMDPs, obtaining performant policies under limited planning time remains a major challenge due to the curse of dimensionality and the curse of history. For many POMDP problems, the value of information (VOI) - the expected performance gain from reasoning about observations - varies over the belief space. We introduce a dynamic programming framework that exploits this structure by conditionally processing observations based on the value of information at each belief. Building on this framework, we propose Value of Information Monte Carlo planning (VOIMCP), a Monte Carlo Tree Search algorithm that allocates computational effort more efficiently by selectively disregarding observation information when the VOI is low, avoiding unnecessary branching of observations. We provide theoretical guarantees on the near-optimality of our VOI reasoning framework and derive non-asymptotic convergence bounds for VOIMCP. Simulation evaluations demonstrate that VOIMCP outperforms baselines on several POMDP benchmarks.