Missingness as Stability: Understanding the Structure of Missingness in Longitudinal EHR data and its Impact on Reinforcement Learning in Healthcare
This work tackles a specific data preprocessing problem in healthcare reinforcement learning, offering an incremental improvement for researchers and practitioners in this domain.
The paper addresses the issue of missing data in longitudinal EHRs for reinforcement learning in healthcare, where common practices like LOCF imputation discard missingness information, and proposes an alternative patient state representation that incorporates missingness, resulting in consistently better performance for optimal control as measured by off-policy policy evaluation.
There is an emerging trend in the reinforcement learning for healthcare literature. In order to prepare longitudinal, irregularly sampled, clinical datasets for reinforcement learning algorithms, many researchers will resample the time series data to short, regular intervals and use last-observation-carried-forward (LOCF) imputation to fill in these gaps. Typically, they will not maintain any explicit information about which values were imputed. In this work, we (1) call attention to this practice and discuss its potential implications; (2) propose an alternative representation of the patient state that addresses some of these issues; and (3) demonstrate in a novel but representative clinical dataset that our alternative representation yields consistently better results for achieving optimal control, as measured by off-policy policy evaluation, compared to representations that do not incorporate missingness information.