Quantifying the Variability Collapse of Neural Networks
This work addresses the need for better metrics to understand neural network behavior, particularly in transfer learning, but is incremental as it builds on the established Neural Collapse paradigm.
The paper tackles the problem of quantifying variability collapse in neural networks by proposing a new metric called Variability Collapse Index (VCI), which is shown to be indicative of transferability and enjoys theoretical properties like invariance under linear transformations.
Recent studies empirically demonstrate the positive relationship between the transferability of neural networks and the within-class variation of the last layer features. The recently discovered Neural Collapse (NC) phenomenon provides a new perspective of understanding such last layer geometry of neural networks. In this paper, we propose a novel metric, named Variability Collapse Index (VCI), to quantify the variability collapse phenomenon in the NC paradigm. The VCI metric is well-motivated and intrinsically related to the linear probing loss on the last layer features. Moreover, it enjoys desired theoretical and empirical properties, including invariance under invertible linear transformations and numerical stability, that distinguishes it from previous metrics. Our experiments verify that VCI is indicative of the variability collapse and the transferability of pretrained neural networks.