Model updating after interventions paradoxically introduces bias
This addresses a critical issue for practitioners in fields like healthcare and finance where model updates can lead to unintended consequences, though it is incremental as it builds on existing causal frameworks.
The authors tackled the problem of bias introduced when updating predictive models that are already used to drive interventions, showing that naive updating can cause models to converge to predicting their own effect or oscillate, and that improved model-fitting may paradoxically worsen performance.
Machine learning is increasingly being used to generate prediction models for use in a number of real-world settings, from credit risk assessment to clinical decision support. Recent discussions have highlighted potential problems in the updating of a predictive score for a binary outcome when an existing predictive score forms part of the standard workflow, driving interventions. In this setting, the existing score induces an additional causative pathway which leads to miscalibration when the original score is replaced. We propose a general causal framework to describe and address this problem, and demonstrate an equivalent formulation as a partially observed Markov decision process. We use this model to demonstrate the impact of such `naive updating' when performed repeatedly. Namely, we show that successive predictive scores may converge to a point where they predict their own effect, or may eventually tend toward a stable oscillation between two values, and we argue that neither outcome is desirable. Furthermore, we demonstrate that even if model-fitting procedures improve, actual performance may worsen. We complement these findings with a discussion of several potential routes to overcome these issues.