Transfer Learning for Linear Regression: a Statistical Test of Gain
This work addresses a theoretical gap for practitioners in regression tasks, though it is incremental as it builds on existing transfer learning concepts.
The paper tackles the lack of theoretical results for transfer learning in regression by proposing a framework for parameter transfer in linear models, showing that transfer quality depends on an eigenbasis representation and constructing a statistical test to predict risk reduction, with efficiency demonstrated on synthetic and real electricity data.
Transfer learning, also referred as knowledge transfer, aims at reusing knowledge from a source dataset to a similar target one. While many empirical studies illustrate the benefits of transfer learning, few theoretical results are established especially for regression problems. In this paper a theoretical framework for the problem of parameter transfer for the linear model is proposed. It is shown that the quality of transfer for a new input vector $x$ depends on its representation in an eigenbasis involving the parameters of the problem. Furthermore a statistical test is constructed to predict whether a fine-tuned model has a lower prediction quadratic risk than the base target model for an unobserved sample. Efficiency of the test is illustrated on synthetic data as well as real electricity consumption data.