Phase Transitions in Transfer Learning for High-Dimensional Perceptrons
This work provides theoretical insights into the conditions under which transfer learning is beneficial, particularly for researchers and practitioners dealing with high-dimensional perceptron-like models.
This paper theoretically analyzes transfer learning using a simplified perceptron model, revealing a phase transition where transfer shifts from detrimental (negative transfer) to beneficial (positive transfer) as the similarity between source and target tasks increases past a specific threshold.
Transfer learning seeks to improve the generalization performance of a target task by exploiting the knowledge learned from a related source task. Central questions include deciding what information one should transfer and when transfer can be beneficial. The latter question is related to the so-called negative transfer phenomenon, where the transferred source information actually reduces the generalization performance of the target task. This happens when the two tasks are sufficiently dissimilar. In this paper, we present a theoretical analysis of transfer learning by studying a pair of related perceptron learning tasks. Despite the simplicity of our model, it reproduces several key phenomena observed in practice. Specifically, our asymptotic analysis reveals a phase transition from negative transfer to positive transfer as the similarity of the two tasks moves past a well-defined threshold.