A unified framework for dataset shift diagnostics
This addresses the challenge of dataset shift for practitioners in machine learning, enabling better adaptation or retraining when labeled target data is limited, though it appears incremental as it builds on existing shift detection concepts.
The paper tackles the problem of dataset shift in supervised learning, which can degrade predictor performance, by proposing a unified framework called DetectShift that quantifies and tests for multiple types of shifts, with experimental results showing its effectiveness in high-dimensional settings.
Supervised learning techniques typically assume training data originates from the target population. Yet, in reality, dataset shift frequently arises, which, if not adequately taken into account, may decrease the performance of their predictors. In this work, we propose a novel and flexible framework called DetectShift that quantifies and tests for multiple dataset shifts, encompassing shifts in the distributions of $(X, Y)$, $X$, $Y$, $X|Y$, and $Y|X$. DetectShift equips practitioners with insights into data shifts, facilitating the adaptation or retraining of predictors using both source and target data. This proves extremely valuable when labeled samples in the target domain are limited. The framework utilizes test statistics with the same nature to quantify the magnitude of the various shifts, making results more interpretable. It is versatile, suitable for regression and classification tasks, and accommodates diverse data forms - tabular, text, or image. Experimental results demonstrate the effectiveness of DetectShift in detecting dataset shifts even in higher dimensions.