Reinforcement Learning with Heterogeneous Data: Estimation and Inference
This addresses the challenge of making reliable decisions in domains like healthcare and business with diverse populations, representing an incremental improvement by adapting RL to heterogeneity.
The authors tackled the problem of reinforcement learning with heterogeneous populations, where classical methods focusing on mean returns can be misleading, by introducing the K-Heterogeneous MDP and auto-clustered algorithms that automatically detect sub-populations and estimate policies for each, showing advantages in an empirical study on the MIMIC-III dataset.
Reinforcement Learning (RL) has the promise of providing data-driven support for decision-making in a wide range of problems in healthcare, education, business, and other domains. Classical RL methods focus on the mean of the total return and, thus, may provide misleading results in the setting of the heterogeneous populations that commonly underlie large-scale datasets. We introduce the K-Heterogeneous Markov Decision Process (K-Hetero MDP) to address sequential decision problems with population heterogeneity. We propose the Auto-Clustered Policy Evaluation (ACPE) for estimating the value of a given policy, and the Auto-Clustered Policy Iteration (ACPI) for estimating the optimal policy in a given policy class. Our auto-clustered algorithms can automatically detect and identify homogeneous sub-populations, while estimating the Q function and the optimal policy for each sub-population. We establish convergence rates and construct confidence intervals for the estimators obtained by the ACPE and ACPI. We present simulations to support our theoretical findings, and we conduct an empirical study on the standard MIMIC-III dataset. The latter analysis shows evidence of value heterogeneity and confirms the advantages of our new method.