Evaluating covariate balance for long time horizon Markov decision processes
For researchers applying offline RL to treatment recommendations, this work highlights methodological weaknesses and calls for improved diagnostics.
This paper investigates whether covariate balance diagnostics can detect hidden confounding in offline reinforcement learning studies for treatment recommendations, finding that existing studies are not statistically robust due to high bias risk or insufficient metrics.
This article explores the application of covariate balance diagnostics for detecting the presence of hidden confounding/model miss-specification in studies applying offline reinforcement learning (RL) to deriving optimal treatment recommendations. The results demonstrate that, either there is a high risk of bias within existing offline RL studies for treatment recommendations or, existing covariate balance metrics are not sufficient to assess such studies. Regardless, existing offline RL studies cannot be concluded as being statistically robust. The conclusions propose future research directions for obtaining more methodologically robust applications of offline RL to treatment recommendation problems.