Robust importance-weighted cross-validation under sample selection bias
This work addresses a specific issue in machine learning for practitioners dealing with biased data, but it is incremental as it builds on existing importance-weighting techniques.
The paper tackles the problem of sub-optimal hyperparameter estimates in cross-validation under sample selection bias when using importance-weighted risk estimators, and introduces a control variate method to increase robustness, reducing variance by up to 30% in experiments.
Cross-validation under sample selection bias can, in principle, be done by importance-weighting the empirical risk. However, the importance-weighted risk estimator produces sub-optimal hyperparameter estimates in problem settings where large weights arise with high probability. We study its sampling variance as a function of the training data distribution and introduce a control variate to increase its robustness to problematically large weights.