HOPE: Human-Centric Off-Policy Evaluation for E-Learning and Healthcare
This addresses the problem of safely evaluating reinforcement learning policies in high-stakes human-centric applications, representing an incremental improvement with domain-specific focus.
The paper tackles the challenge of off-policy evaluation in human-centric environments like e-learning and healthcare, where states are partially observable and rewards are aggregated, by proposing HOPE to reconstruct immediate rewards and estimate returns, achieving reliable predictions and outperforming state-of-the-art benchmarks in experiments on sepsis treatments and intelligent tutoring systems.
Reinforcement learning (RL) has been extensively researched for enhancing human-environment interactions in various human-centric tasks, including e-learning and healthcare. Since deploying and evaluating policies online are high-stakes in such tasks, off-policy evaluation (OPE) is crucial for inducing effective policies. In human-centric environments, however, OPE is challenging because the underlying state is often unobservable, while only aggregate rewards can be observed (students' test scores or whether a patient is released from the hospital eventually). In this work, we propose a human-centric OPE (HOPE) to handle partial observability and aggregated rewards in such environments. Specifically, we reconstruct immediate rewards from the aggregated rewards considering partial observability to estimate expected total returns. We provide a theoretical bound for the proposed method, and we have conducted extensive experiments in real-world human-centric tasks, including sepsis treatments and an intelligent tutoring system. Our approach reliably predicts the returns of different policies and outperforms state-of-the-art benchmarks using both standard validation methods and human-centric significance tests.