Understand the Effect of Importance Weighting in Deep Learning on Dataset Shift
This work questions the practical utility of importance weighting for real-world distribution shifts, which is an incremental finding for researchers in robust machine learning.
The study evaluated importance weighting in deep neural networks under dataset shifts, finding that its effects fade with prolonged training and it yields no significant performance gains on complex data like CIFAR-10.
We evaluate the effectiveness of importance weighting in deep neural networks under label shift and covariate shift. On synthetic 2D data (linearly separable and moon-shaped) using logistic regression and MLPs, we observe that weighting strongly affects decision boundaries early in training but fades with prolonged optimization. On CIFAR-10 with various class imbalances, only L2 regularization (not dropout) helps preserve weighting effects. In a covariate-shift experiment, importance weighting yields no significant performance gain, highlighting challenges on complex data. Our results call into question the practical utility of importance weighting for real-world distribution shifts.