OCMDP: Observation-Constrained Markov Decision Process
This addresses cost-sensitive decision-making for applications like healthcare, though it appears incremental as it builds on existing MDP frameworks with a novel constraint.
The paper tackles the problem of balancing information acquisition costs with decision-making benefits in environments where observations are expensive, by introducing the Observation-Constrained Markov Decision Process (OCMDP) and a model-free deep reinforcement learning algorithm, achieving a substantial reduction in observation costs and outperforming baseline methods in efficiency.
In many practical applications, decision-making processes must balance the costs of acquiring information with the benefits it provides. Traditional control systems often assume full observability, an unrealistic assumption when observations are expensive. We tackle the challenge of simultaneously learning observation and control strategies in such cost-sensitive environments by introducing the Observation-Constrained Markov Decision Process (OCMDP), where the policy influences the observability of the true state. To manage the complexity arising from the combined observation and control actions, we develop an iterative, model-free deep reinforcement learning algorithm that separates the sensing and control components of the policy. This decomposition enables efficient learning in the expanded action space by focusing on when and what to observe, as well as determining optimal control actions, without requiring knowledge of the environment's dynamics. We validate our approach on a simulated diagnostic task and a realistic healthcare environment using HeartPole. Given both scenarios, the experimental results demonstrate that our model achieves a substantial reduction in observation costs on average, significantly outperforming baseline methods by a notable margin in efficiency.