Score-Based Change Detection for Gradient-Based Learning Machines
This provides an incremental solution for practitioners needing to monitor and control ML models in evolving data streams.
The paper tackles the problem of monitoring machine learning models for changes over time by introducing a generic score-based change detection method for models trained via empirical risk minimization, establishing test consistency and calibration for false alarm control, and demonstrating its versatility on synthetic and real data.
The widespread use of machine learning algorithms calls for automatic change detection algorithms to monitor their behavior over time. As a machine learning algorithm learns from a continuous, possibly evolving, stream of data, it is desirable and often critical to supplement it with a companion change detection algorithm to facilitate its monitoring and control. We present a generic score-based change detection method that can detect a change in any number of components of a machine learning model trained via empirical risk minimization. This proposed statistical hypothesis test can be readily implemented for such models designed within a differentiable programming framework. We establish the consistency of the hypothesis test and show how to calibrate it to achieve a prescribed false alarm rate. We illustrate the versatility of the approach on synthetic and real data.