k-NN as a Simple and Effective Estimator of Transferability
This addresses the challenge for machine learning practitioners needing reliable transferability estimates, though it is incremental as it adapts an existing method to a new application.
The paper tackled the problem of predicting transfer learning performance across domain, task, and architecture shifts by evaluating 23 metrics over 42,000 experiments, finding that none performed well overall, but a simple k-nearest neighbor estimator surpassed existing metrics in accuracy, efficiency, and ease of use.
How well can one expect transfer learning to work in a new setting where the domain is shifted, the task is different, and the architecture changes? Many transfer learning metrics have been proposed to answer this question. But how accurate are their predictions in a realistic new setting? We conducted an extensive evaluation involving over 42,000 experiments comparing 23 transferability metrics across 16 different datasets to assess their ability to predict transfer performance. Our findings reveal that none of the existing metrics perform well across the board. However, we find that a simple k-nearest neighbor evaluation -- as is commonly used to evaluate feature quality for self-supervision -- not only surpasses existing metrics, but also offers better computational efficiency and ease of implementation.