Distributional Shift Adaptation using Domain-Specific Features
This addresses the challenge of distributional shift in open-world scenarios for machine learning practitioners, offering an incremental improvement over prior methods that focus only on invariant features.
The paper tackles the problem of machine learning algorithms failing on out-of-distribution data by proposing an approach that uses domain-specific features, including both invariant and target-unique features, to adapt models to new domains. Empirical results show performance improvements of 10-20% over state-of-the-art methods on benchmark datasets.
Machine learning algorithms typically assume that the training and test samples come from the same distributions, i.e., in-distribution. However, in open-world scenarios, streaming big data can be Out-Of-Distribution (OOD), rendering these algorithms ineffective. Prior solutions to the OOD challenge seek to identify invariant features across different training domains. The underlying assumption is that these invariant features should also work reasonably well in the unlabeled target domain. By contrast, this work is interested in the domain-specific features that include both invariant features and features unique to the target domain. We propose a simple yet effective approach that relies on correlations in general regardless of whether the features are invariant or not. Our approach uses the most confidently predicted samples identified by an OOD base model (teacher model) to train a new model (student model) that effectively adapts to the target domain. Empirical evaluations on benchmark datasets show that the performance is improved over the SOTA by ~10-20%