The Impact of Geometric Complexity on Neural Collapse in Transfer Learning
This work provides incremental theoretical insights into transfer learning mechanisms for machine learning researchers, focusing on geometric complexity and neural collapse.
The paper tackles the problem of understanding transfer learning's success by linking geometric complexity of learned representations to neural collapse, showing that mechanisms affecting geometric complexity influence neural collapse and improve downstream task performance, particularly in few-shot settings.
Many of the recent remarkable advances in computer vision and language models can be attributed to the success of transfer learning via the pre-training of large foundation models. However, a theoretical framework which explains this empirical success is incomplete and remains an active area of research. Flatness of the loss surface and neural collapse have recently emerged as useful pre-training metrics which shed light on the implicit biases underlying pre-training. In this paper, we explore the geometric complexity of a model's learned representations as a fundamental mechanism that relates these two concepts. We show through experiments and theory that mechanisms which affect the geometric complexity of the pre-trained network also influence the neural collapse. Furthermore, we show how this effect of the geometric complexity generalizes to the neural collapse of new classes as well, thus encouraging better performance on downstream tasks, particularly in the few-shot setting.