Prominent Roles of Conditionally Invariant Components in Domain Adaptation: Theory and Algorithms
This work addresses domain adaptation for machine learning practitioners by clarifying assumptions and improving algorithms, though it is incremental as it builds on existing methods like DIP.
The paper tackles the problem of domain adaptation by focusing on conditionally invariant components (CICs), showing that they provide target risk guarantees and improve algorithms like domain invariant projection (DIP) to handle failure scenarios such as label-flipping features, with validation on datasets including MNIST, CelebA, Camelyon17, and DomainNet.
Domain adaptation (DA) is a statistical learning problem that arises when the distribution of the source data used to train a model differs from that of the target data used to evaluate the model. While many DA algorithms have demonstrated considerable empirical success, blindly applying these algorithms can often lead to worse performance on new datasets. To address this, it is crucial to clarify the assumptions under which a DA algorithm has good target performance. In this work, we focus on the assumption of the presence of conditionally invariant components (CICs), which are relevant for prediction and remain conditionally invariant across the source and target data. We demonstrate that CICs, which can be estimated through conditional invariant penalty (CIP), play three prominent roles in providing target risk guarantees in DA. First, we propose a new algorithm based on CICs, importance-weighted conditional invariant penalty (IW-CIP), which has target risk guarantees beyond simple settings such as covariate shift and label shift. Second, we show that CICs help identify large discrepancies between source and target risks of other DA algorithms. Finally, we demonstrate that incorporating CICs into the domain invariant projection (DIP) algorithm can address its failure scenario caused by label-flipping features. We support our new algorithms and theoretical findings via numerical experiments on synthetic data, MNIST, CelebA, Camelyon17, and DomainNet datasets.