Concept Drift and Covariate Shift Detection Ensemble with Lagged Labels
This work is significant for practitioners in model serving who face performance degradation due to evolving data distributions and delayed ground truth, offering a more robust and automated solution for drift detection and model retraining.
The paper addresses the problem of concept drift and covariate shift in model serving, where data distribution changes over time, leading to degraded model performance. The authors propose a method that uses six different signals beyond classification error, handles lagged labels, and automatically selects retraining data, outperforming state-of-the-art methods across various data types and changes.
In model serving, having one fixed model during the entire often life-long inference process is usually detrimental to model performance, as data distribution evolves over time, resulting in lack of reliability of the model trained on historical data. It is important to detect changes and retrain the model in time. The existing methods generally have three weaknesses: 1) using only classification error rate as signal, 2) assuming ground truth labels are immediately available after features from samples are received and 3) unable to decide what data to use to retrain the model when change occurs. We address the first problem by utilizing six different signals to capture a wide range of characteristics of data, and we address the second problem by allowing lag of labels, where labels of corresponding features are received after a lag in time. For the third problem, our proposed method automatically decides what data to use to retrain based on the signals. Extensive experiments on structured and unstructured data for different type of data changes establish that our method consistently outperforms the state-of-the-art methods by a large margin.