Towards Understanding Feature Learning in Parameter Transfer
This work addresses a theoretical gap in transfer learning for researchers, providing insights into parameter reuse conditions, but it is incremental as it builds on existing paradigms without introducing new methods.
The paper tackles the lack of theoretical understanding of when partial parameter transfer in transfer learning is beneficial, analyzing ReLU CNNs to characterize how inherited parameters carry universal knowledge and identify factors that amplify their impact, with empirical validation showing cases where transfer can lower test accuracy compared to training from scratch.
Parameter transfer is a central paradigm in transfer learning, enabling knowledge reuse across tasks and domains by sharing model parameters between upstream and downstream models. However, when only a subset of parameters from the upstream model is transferred to the downstream model, there remains a lack of theoretical understanding of the conditions under which such partial parameter reuse is beneficial and of the factors that govern its effectiveness. To address this gap, we analyze a setting in which both the upstream and downstream models are ReLU convolutional neural networks (CNNs). Within this theoretical framework, we characterize how the inherited parameters act as carriers of universal knowledge and identify key factors that amplify their beneficial impact on the target task. Furthermore, our analysis provides insight into why, in certain cases, transferring parameters can lead to lower test accuracy on the target task than training a new model from scratch. Numerical experiments and real-world data experiments are conducted to empirically validate our theoretical findings.