Occam's model: Selecting simpler representations for better transferability estimation
This work addresses the problem of selecting the most suitable pre-trained model for a given task, which is crucial for practitioners and researchers in the machine learning community, particularly those using online model repositories.
The authors tackled the problem of transferability estimation in pre-trained models, achieving a 32% increase in Kendall's Tau compared to state-of-the-art baselines. Their approach provides a unique perspective on transferability estimation by viewing it as a measure of how easily a pre-trained model's representations can be trained to separate target classes.
Fine-tuning models that have been pre-trained on large datasets has become a cornerstone of modern machine learning workflows. With the widespread availability of online model repositories, such as Hugging Face, it is now easier than ever to fine-tune pre-trained models for specific tasks. This raises a critical question: which pre-trained model is most suitable for a given task? This problem is called transferability estimation. In this work, we introduce two novel and effective metrics for estimating the transferability of pre-trained models. Our approach is grounded in viewing transferability as a measure of how easily a pre-trained model's representations can be trained to separate target classes, providing a unique perspective on transferability estimation. We rigorously evaluate the proposed metrics against state-of-the-art alternatives across diverse problem settings, demonstrating their robustness and practical utility. Additionally, we present theoretical insights that explain our metrics' efficacy and adaptability to various scenarios. We experimentally show that our metrics increase Kendall's Tau by up to 32% compared to the state-of-the-art baselines.