Efficient Hyperparameter Optimization under Multi-Source Covariate Shift
This addresses the problem of efficient hyperparameter tuning for machine learning practitioners when data distributions shift between training and testing, though it is incremental as it builds on existing covariate shift and optimization methods.
The paper tackles hyperparameter optimization under multi-source covariate shift, where labeled data is available only from source tasks and unlabeled data from a target task, by proposing a variance-reduced estimator to approximate the target objective, resulting in a tractable procedure with no-regret guarantees that broadens applications of automated hyperparameter optimization.
A typical assumption in supervised machine learning is that the train (source) and test (target) datasets follow completely the same distribution. This assumption is, however, often violated in uncertain real-world applications, which motivates the study of learning under covariate shift. In this setting, the naive use of adaptive hyperparameter optimization methods such as Bayesian optimization does not work as desired since it does not address the distributional shift among different datasets. In this work, we consider a novel hyperparameter optimization problem under the multi-source covariate shift whose goal is to find the optimal hyperparameters for a target task of interest using only unlabeled data in a target task and labeled data in multiple source tasks. To conduct efficient hyperparameter optimization for the target task, it is essential to estimate the target objective using only the available information. To this end, we construct the variance reduced estimator that unbiasedly approximates the target objective with a desirable variance property. Building on the proposed estimator, we provide a general and tractable hyperparameter optimization procedure, which works preferably in our setting with a no-regret guarantee. The experiments demonstrate that the proposed framework broadens the applications of automated hyperparameter optimization.