Partially Observable Reference Policy Programming: Solving POMDPs Sans Numerical Optimisation
This addresses the challenge of efficient online planning in POMDPs for applications like robotics or emergency scenarios, representing a novel method rather than an incremental improvement.
The paper tackles the problem of solving partially observable Markov decision processes (POMDPs) without numerical optimization by proposing Partially Observable Reference Policy Programming, which samples future histories deeply while gradually updating policies. The result shows theoretical guarantees with bounded performance loss based on average sampling errors and empirical outperformance of current online benchmarks in large-scale problems like a helicopter emergency scenario requiring ~150 planning steps.
This paper proposes Partially Observable Reference Policy Programming, a novel anytime online approximate POMDP solver which samples meaningful future histories very deeply while simultaneously forcing a gradual policy update. We provide theoretical guarantees for the algorithm's underlying scheme which say that the performance loss is bounded by the average of the sampling approximation errors rather than the usual maximum, a crucial requirement given the sampling sparsity of online planning. Empirical evaluations on two large-scale problems with dynamically evolving environments -- including a helicopter emergency scenario in the Corsica region requiring approximately 150 planning steps -- corroborate the theoretical results and indicate that our solver considerably outperforms current online benchmarks.