Ensembling Shift Detectors: an Extensive Empirical Evaluation
This work addresses the challenge of unknown shift types in real-life machine learning deployments, offering a more reliable detection method.
The paper tackled the problem of dataset shift detection by proposing an ensembling technique that combines multiple shift detectors and tunes their statistical significance levels, achieving robust detection across various shift types. The approach was validated through a large-scale benchmark study on synthetic shifts applied to real-world datasets.
The term dataset shift refers to the situation where the data used to train a machine learning model is different from where the model operates. While several types of shifts naturally occur, existing shift detectors are usually designed to address only a specific type of shift. We propose a simple yet powerful technique to ensemble complementary shift detectors, while tuning the significance level of each detector's statistical test to the dataset. This enables a more robust shift detection, capable of addressing all different types of shift, which is essential in real-life settings where the precise shift type is often unknown. This approach is validated by a large-scale statistically sound benchmark study over various synthetic shifts applied to real-world structured datasets.