Diverse Imagenet Models Transfer Better
This work addresses the need for better transfer learning models in computer vision, offering an incremental improvement by identifying and optimizing for feature diversity alongside accuracy.
The paper tackles the problem that ImageNet accuracy alone does not guarantee good transfer to downstream tasks, showing that high feature diversity jointly with accuracy promotes transferability, and proposes a method combining self-supervised and supervised pretraining to achieve this, demonstrating improved results on multiple architectures and tasks.
A commonly accepted hypothesis is that models with higher accuracy on Imagenet perform better on other downstream tasks, leading to much research dedicated to optimizing Imagenet accuracy. Recently this hypothesis has been challenged by evidence showing that self-supervised models transfer better than their supervised counterparts, despite their inferior Imagenet accuracy. This calls for identifying the additional factors, on top of Imagenet accuracy, that make models transferable. In this work we show that high diversity of the features learnt by the model promotes transferability jointly with Imagenet accuracy. Encouraged by the recent transferability results of self-supervised models, we propose a method that combines self-supervised and supervised pretraining to generate models with both high diversity and high accuracy, and as a result high transferability. We demonstrate our results on several architectures and multiple downstream tasks, including both single-label and multi-label classification.