Probing transfer learning with a model of synthetic correlated datasets
This work provides theoretical insights into transfer learning for machine learning researchers, but it is incremental as it builds on existing models without introducing a new paradigm.
The authors tackled the limited theoretical understanding of transfer learning by developing a solvable model of synthetic correlated datasets to analytically characterize generalization performance when transferring learned features from a source to a target task. They showed that their model captures key features of real-data transfer learning and systematically identified conditions under which feature transfer improves generalization.
Transfer learning can significantly improve the sample efficiency of neural networks, by exploiting the relatedness between a data-scarce target task and a data-abundant source task. Despite years of successful applications, transfer learning practice often relies on ad-hoc solutions, while theoretical understanding of these procedures is still limited. In the present work, we re-think a solvable model of synthetic data as a framework for modeling correlation between data-sets. This setup allows for an analytic characterization of the generalization performance obtained when transferring the learned feature map from the source to the target task. Focusing on the problem of training two-layer networks in a binary classification setting, we show that our model can capture a range of salient features of transfer learning with real data. Moreover, by exploiting parametric control over the correlation between the two data-sets, we systematically investigate under which conditions the transfer of features is beneficial for generalization.