Quantifying Potential Observation Missingness in Inverse Reinforcement Learning
For researchers and practitioners using IRL on real-world behavioral data, this work highlights a critical but overlooked issue and provides a tool to assess data quality, though the approach is incremental.
The paper identifies and addresses the problem of missing observations in inverse reinforcement learning (IRL) datasets, which can cause expert actions to appear suboptimal. The authors develop an algorithm to quantify the minimal perturbations needed for expert actions to appear optimal, demonstrating its utility on synthetic and real-world healthcare datasets.
Inverse reinforcement learning (IRL), which infers reward functions from demonstrations, is a valuable tool for modeling and understanding decision-making behavior. Many variants of IRL have been developed to capture complexities of human decision-making, such as subjective beliefs, imperfect planning, and dynamic goals. However, an often-overlooked issue in real-world behavioral datasets is that the recorded data may be missing observations that were available to the original decision-maker. In use-inspired settings such as healthcare, this can make expert actions appear suboptimal, even when they were near-optimal given the information available at the time. As a result, the rewards learned by standard IRL may be misleading. In this paper, we identify the minimal perturbations to the recorded observations needed for the expert's actions to appear optimal. We develop a practical algorithm for this problem and demonstrate its utility for quantifying the possible extent of missing observations in behavioral datasets through extensive experiments on synthetic navigation tasks, a cancer treatment simulator, and ICU treatment data.