PACTran: PAC-Bayesian Metrics for Estimating the Transferability of Pretrained Models to Classification Tasks
This addresses the need for better pretrained model selection in transfer learning, offering a theoretically motivated improvement over heuristic approaches.
The paper tackles the problem of selecting the best pretrained model for downstream classification tasks by introducing PACTran, a theoretically grounded family of metrics derived from PAC-Bayesian bounds, and shows it is more consistent and effective than existing methods in evaluations on vision and language-and-vision tasks.
With the increasing abundance of pretrained models in recent years, the problem of selecting the best pretrained checkpoint for a particular downstream classification task has been gaining increased attention. Although several methods have recently been proposed to tackle the selection problem (e.g. LEEP, H-score), these methods resort to applying heuristics that are not well motivated by learning theory. In this paper we present PACTran, a theoretically grounded family of metrics for pretrained model selection and transferability measurement. We first show how to derive PACTran metrics from the optimal PAC-Bayesian bound under the transfer learning setting. We then empirically evaluate three metric instantiations of PACTran on a number of vision tasks (VTAB) as well as a language-and-vision (OKVQA) task. An analysis of the results shows PACTran is a more consistent and effective transferability measure compared to existing selection methods.