Understanding Transfer Learning via Mean-field Analysis
This work provides theoretical insights into transfer learning, which is important for machine learning practitioners seeking to improve model generalization, though it appears incremental as it builds on existing mean-field analysis methods.
The paper tackles the problem of understanding generalization errors in transfer learning by proposing a novel framework using mean-field analysis and differential calculus on probability measures, establishing conditions for generalization error and population risk convergence rates in two transfer learning scenarios and demonstrating benefits for one-hidden-layer neural networks under specific assumptions.
We propose a novel framework for exploring generalization errors of transfer learning through the lens of differential calculus on the space of probability measures. In particular, we consider two main transfer learning scenarios, $α$-ERM and fine-tuning with the KL-regularized empirical risk minimization and establish generic conditions under which the generalization error and the population risk convergence rates for these scenarios are studied. Based on our theoretical results, we show the benefits of transfer learning with a one-hidden-layer neural network in the mean-field regime under some suitable integrability and regularity assumptions on the loss and activation functions.