Robust Finite-Memory Policy Gradients for Hidden-Model POMDPs
This addresses the need for robust decision-making in uncertain environments for applications like robotics or autonomous systems, but it is incremental as it builds on existing POMDP and robustness techniques.
The paper tackles the problem of computing robust policies for hidden-model POMDPs, which are sets of environment models with uncertainty about the true model, by combining formal verification for worst-case evaluation and subgradient ascent for optimization, resulting in policies that are more robust and generalize better to unseen environments while scaling to over a hundred thousand environments.
Partially observable Markov decision processes (POMDPs) model specific environments in sequential decision-making under uncertainty. Critically, optimal policies for POMDPs may not be robust against perturbations in the environment. Hidden-model POMDPs (HM-POMDPs) capture sets of different environment models, that is, POMDPs with a shared action and observation space. The intuition is that the true model is hidden among a set of potential models, and it is unknown which model will be the environment at execution time. A policy is robust for a given HM-POMDP if it achieves sufficient performance for each of its POMDPs. We compute such robust policies by combining two orthogonal techniques: (1) a deductive formal verification technique that supports tractable robust policy evaluation by computing a worst-case POMDP within the HM-POMDP, and (2) subgradient ascent to optimize the candidate policy for a worst-case POMDP. The empirical evaluation shows that, compared to various baselines, our approach (1) produces policies that are more robust and generalize better to unseen POMDPs, and (2) scales to HM-POMDPs that consist of over a hundred thousand environments.