Where to start? Analyzing the potential value of intermediate models
This provides a practical approach for selecting base models in real-world settings, though it is incremental as it builds on existing observations about finetuning.
The study systematically analyzes the intertraining scheme, where finetuned models serve as better base models for new tasks, across English classification tasks, finding that intertraining gain can be independently assessed for target datasets and base models, contrary to the belief that source-target alignment is key.
Previous studies observed that finetuned models may be better base models than the vanilla pretrained model. Such a model, finetuned on some source dataset, may provide a better starting point for a new finetuning process on a desired target dataset. Here, we perform a systematic analysis of this intertraining scheme, over a wide range of English classification tasks. Surprisingly, our analysis suggests that the potential intertraining gain can be analyzed independently for the target dataset under consideration, and for a base model being considered as a starting point. This is in contrast to current perception that the alignment between the target dataset and the source dataset used to generate the base model is a major factor in determining intertraining success. We analyze different aspects that contribute to each. Furthermore, we leverage our analysis to propose a practical and efficient approach to determine if and how to select a base model in real-world settings. Last, we release an updating ranking of best models in the HuggingFace hub per architecture https://ibm.github.io/model-recycling/.