Optimizing importance weighting in the presence of sub-population shifts
This work addresses performance degradation in machine learning models due to sub-population shifts, offering an incremental improvement over existing weighting heuristics.
The paper tackles the problem of distribution shift between training and test data by optimizing importance weighting to account for bias-variance trade-offs, resulting in significantly improved generalization performance for deep neural networks under sub-population shifts.
A distribution shift between the training and test data can severely harm performance of machine learning models. Importance weighting addresses this issue by assigning different weights to data points during training. We argue that existing heuristics for determining the weights are suboptimal, as they neglect the increase of the variance of the estimated model due to the finite sample size of the training data. We interpret the optimal weights in terms of a bias-variance trade-off, and propose a bi-level optimization procedure in which the weights and model parameters are optimized simultaneously. We apply this optimization to existing importance weighting techniques for last-layer retraining of deep neural networks in the presence of sub-population shifts and show empirically that optimizing weights significantly improves generalization performance.