Multi-Environment POMDPs: Discrete Model Uncertainty Under Partial Observability
This work addresses robustness in AI planning for scenarios where multiple domain experts disagree on models, which is an incremental extension of existing POMDP frameworks.
The paper tackles the problem of decision-making under partial observability with discrete model uncertainty by introducing Multi-Environment POMDPs (ME-POMDPs) and adversarial-belief POMDPs (AB-POMDPs), and it presents exact and approximate algorithms that compute robust policies for standard POMDP benchmarks extended to this setting.
Multi-environment POMDPs (ME-POMDPs) extend standard POMDPs with discrete model uncertainty. ME-POMDPs represent a finite set of POMDPs that share the same state, action, and observation spaces, but may arbitrarily vary in their transition, observation, and reward models. Such models arise, for instance, when multiple domain experts disagree on how to model a problem. The goal is to find a single policy that is robust against any choice of POMDP within the set, i.e., a policy that maximizes the worst-case reward across all POMDPs. We generalize and expand on existing work in the following way. First, we show that ME-POMDPs can be generalized to POMDPs with sets of initial beliefs, which we call adversarial-belief POMDPs (AB-POMDPs). Second, we show that any arbitrary ME-POMDP can be reduced to a ME-POMDP that only varies in its transition and reward functions or only in its observation and reward functions, while preserving (optimal) policies. We then devise exact and approximate (point-based) algorithms to compute robust policies for AB-POMDPs, and thus ME-POMDPs. We demonstrate that we can compute policies for standard POMDP benchmarks extended to the multi-environment setting.