Unleashing the power of Neural Collapse for Transferability Estimation
This work addresses the need for efficient heuristics to estimate model transferability without fine-tuning, which is incremental as it builds on prior neural collapse insights.
The paper tackled the problem of transferability estimation by proposing a method called Fair Collapse (FaCe) that measures neural collapse in pre-trained models, achieving state-of-the-art performance across tasks like image classification, semantic segmentation, and text classification.
Transferability estimation aims to provide heuristics for quantifying how suitable a pre-trained model is for a specific downstream task, without fine-tuning them all. Prior studies have revealed that well-trained models exhibit the phenomenon of Neural Collapse. Based on a widely used neural collapse metric in existing literature, we observe a strong correlation between the neural collapse of pre-trained models and their corresponding fine-tuned models. Inspired by this observation, we propose a novel method termed Fair Collapse (FaCe) for transferability estimation by comprehensively measuring the degree of neural collapse in the pre-trained model. Typically, FaCe comprises two different terms: the variance collapse term, which assesses the class separation and within-class compactness, and the class fairness term, which quantifies the fairness of the pre-trained model towards each class. We investigate FaCe on a variety of pre-trained classification models across different network architectures, source datasets, and training loss functions. Results show that FaCe yields state-of-the-art performance on different tasks including image classification, semantic segmentation, and text classification, which demonstrate the effectiveness and generalization of our method.