Out of the Ordinary: Spectrally Adapting Regression for Covariate Shift
This addresses a relatively unexplored problem in machine learning for researchers and practitioners needing robust regression models under distribution shifts.
The paper tackles out-of-distribution generalization for regression by analyzing how Ordinary Least Squares is sensitive to covariate shift, proposing a spectral adaptation method for neural networks that improves performance on synthetic and real-world datasets.
Designing deep neural network classifiers that perform robustly on distributions differing from the available training data is an active area of machine learning research. However, out-of-distribution generalization for regression-the analogous problem for modeling continuous targets-remains relatively unexplored. To tackle this problem, we return to first principles and analyze how the closed-form solution for Ordinary Least Squares (OLS) regression is sensitive to covariate shift. We characterize the out-of-distribution risk of the OLS model in terms of the eigenspectrum decomposition of the source and target data. We then use this insight to propose a method for adapting the weights of the last layer of a pre-trained neural regression model to perform better on input data originating from a different distribution. We demonstrate how this lightweight spectral adaptation procedure can improve out-of-distribution performance for synthetic and real-world datasets.