A generalized approach to label shift: the Conditional Probability Shift Model
This work addresses distributional shift issues in practical ML applications, such as medical data analysis, by proposing a novel framework that extends beyond traditional covariate and label shift scenarios, though it is incremental in building upon existing shift models.
The paper tackles the problem of distributional shift in machine learning by introducing a new setting called Conditional Probability Shift (CPS), which captures changes in the conditional distribution of the class variable given specific features, and presents the Conditional Probability Shift Model (CPSM) that uses multinomial regression and EM estimation. Experiments on synthetic datasets and the MIMIC medical database show superior balanced classification accuracy on target data compared to existing methods, particularly in cases of conditional distribution shift.
In many practical applications of machine learning, a discrepancy often arises between a source distribution from which labeled training examples are drawn and a target distribution for which only unlabeled data is observed. Traditionally, two main scenarios have been considered to address this issue: covariate shift (CS), where only the marginal distribution of features changes, and label shift (LS), which involves a change in the class variable's prior distribution. However, these frameworks do not encompass all forms of distributional shift. This paper introduces a new setting, Conditional Probability Shift (CPS), which captures the case when the conditional distribution of the class variable given some specific features changes while the distribution of remaining features given the specific features and the class is preserved. For this scenario we present the Conditional Probability Shift Model (CPSM) based on modeling the class variable's conditional probabilities using multinomial regression. Since the class variable is not observed for the target data, the parameters of the multinomial model for its distribution are estimated using the Expectation-Maximization algorithm. The proposed method is generic and can be combined with any probabilistic classifier. The effectiveness of CPSM is demonstrated through experiments on synthetic datasets and a case study using the MIMIC medical database, revealing its superior balanced classification accuracy on the target data compared to existing methods, particularly in situations situations of conditional distribution shift and no apriori distribution shift, which are not detected by LS-based methods.