C-IDS: Solving Contextual POMDP via Information-Directed Objective
This work addresses the challenge of decision-making under uncertainty in environments with hidden dynamics, offering a novel approach for robotics or AI systems, though it appears incremental as it builds on existing POMDP frameworks.
The paper tackles the policy synthesis problem in contextual POMDPs by introducing an information-directed objective that combines reward maximization with uncertainty reduction about latent contexts, resulting in a sublinear Bayesian regret bound and outperforming standard POMDP solvers in faster context identification and higher returns.
We study the policy synthesis problem in contextual partially observable Markov decision processes (CPOMDPs), where the environment is governed by an unknown latent context that induces distinct POMDP dynamics. Our goal is to design a policy that simultaneously maximizes cumulative return and actively reduces uncertainty about the underlying context. We introduce an information-directed objective that augments reward maximization with mutual information between the latent context and the agent's observations. We develop the C-IDS algorithm to synthesize policies that maximize the information-directed objective. We show that the objective can be interpreted as a Lagrangian relaxation of the linear information ratio and prove that the temperature parameter is an upper bound on the information ratio. Based on this characterization, we establish a sublinear Bayesian regret bound over K episodes. We evaluate our approach on a continuous Light-Dark environment and show that it consistently outperforms standard POMDP solvers that treat the unknown context as a latent state variable, achieving faster context identification and higher returns.