Variational Inference for Data-Efficient Model Learning in POMDPs
This addresses the challenge of data-efficient model learning in POMDPs for tasks requiring decision-making under uncertainty, representing an incremental improvement by applying variational inference to a known bottleneck.
The paper tackles the problem of acquiring accurate models for complex partially observable Markov decision processes (POMDPs) by proposing DELIP, an approach using amortized structured variational inference, and shows it leads to effective control strategies when coupled with state-of-the-art planners, particularly in environments with changing or initially unknown reward structures.
Partially observable Markov decision processes (POMDPs) are a powerful abstraction for tasks that require decision making under uncertainty, and capture a wide range of real world tasks. Today, effective planning approaches exist that generate effective strategies given black-box models of a POMDP task. Yet, an open question is how to acquire accurate models for complex domains. In this paper we propose DELIP, an approach to model learning for POMDPs that utilizes amortized structured variational inference. We empirically show that our model leads to effective control strategies when coupled with state-of-the-art planners. Intuitively, model-based approaches should be particularly beneficial in environments with changing reward structures, or where rewards are initially unknown. Our experiments confirm that DELIP is particularly effective in this setting.