Designing monitoring strategies for deployed machine learning algorithms: navigating performativity through a causal lens
This work addresses the challenge of ensuring safety and effectiveness for ML deployment teams by highlighting the complexity of monitoring strategies in performative environments, though it is incremental as it builds on existing causal validation methods.
The paper tackles the problem of monitoring deployed machine learning algorithms in the presence of performativity, where algorithms affect their own data, and proposes using causal inference to systematically choose among monitoring strategies. It demonstrates through a simulation case study on a risk prediction algorithm that different monitoring procedures vary in interpretability, assumptions, and detection speed, impacting real-world system design.
After a machine learning (ML)-based system is deployed, monitoring its performance is important to ensure the safety and effectiveness of the algorithm over time. When an ML algorithm interacts with its environment, the algorithm can affect the data-generating mechanism and be a major source of bias when evaluating its standalone performance, an issue known as performativity. Although prior work has shown how to validate models in the presence of performativity using causal inference techniques, there has been little work on how to monitor models in the presence of performativity. Unlike the setting of model validation, there is much less agreement on which performance metrics to monitor. Different monitoring criteria impact how interpretable the resulting test statistic is, what assumptions are needed for identifiability, and the speed of detection. When this choice is further coupled with the decision to use observational versus interventional data, ML deployment teams are faced with a multitude of monitoring options. The aim of this work is to highlight the relatively under-appreciated complexity of designing a monitoring strategy and how causal reasoning can provide a systematic framework for choosing between these options. As a motivating example, we consider an ML-based risk prediction algorithm for predicting unplanned readmissions. Bringing together tools from causal inference and statistical process control, we consider six monitoring procedures (three candidate monitoring criteria and two data sources) and investigate their operating characteristics in simulation studies. Results from this case study emphasize the seemingly simple (and obvious) fact that not all monitoring systems are created equal, which has real-world impacts on the design and documentation of ML monitoring systems.