Pseudo-Labeling for Kernel Ridge Regression under Covariate Shift
This addresses the problem of covariate shift in regression for machine learning practitioners, offering an incremental improvement through a novel pseudo-labeling approach.
The paper tackles kernel ridge regression under covariate shift by proposing a pseudo-labeling method that uses labeled source data and unlabeled target data to learn a regression function with minimal mean squared error on the target distribution. The result shows the estimator achieves minimax optimal error rates up to a polylogarithmic factor, with pseudo-labeling not significantly hindering performance.
We develop and analyze a principled approach to kernel ridge regression under covariate shift. The goal is to learn a regression function with small mean squared error over a target distribution, based on unlabeled data from there and labeled data that may have a different feature distribution. We propose to split the labeled data into two subsets, and conduct kernel ridge regression on them separately to obtain a collection of candidate models and an imputation model. We use the latter to fill the missing labels and then select the best candidate accordingly. Our non-asymptotic excess risk bounds demonstrate that our estimator adapts effectively to both the structure of the target distribution and the covariate shift. This adaptation is quantified through a notion of effective sample size that reflects the value of labeled source data for the target regression task. Our estimator achieves the minimax optimal error rate up to a polylogarithmic factor, and we find that using pseudo-labels for model selection does not significantly hinder performance.