Foundation Model Insights and a Multi-Model Approach for Superior Fine-Grained One-shot Subset Selection
This addresses the problem of efficient data selection for training in fine-grained image classification, offering a novel method that is incremental over existing foundation model applications.
This work tackled the problem of reducing deep learning training costs by improving one-shot subset selection, finding that foundation models outperform traditional methods on fine-grained datasets and proposing a multi-model approach that achieves state-of-the-art performance on datasets like Oxford-IIIT Pet, Food-101, and Caltech-UCSD Birds-200-2011.
One-shot subset selection serves as an effective tool to reduce deep learning training costs by identifying an informative data subset based on the information extracted by an information extractor (IE). Traditional IEs, typically pre-trained on the target dataset, are inherently dataset-dependent. Foundation models (FMs) offer a promising alternative, potentially mitigating this limitation. This work investigates two key questions: (1) Can FM-based subset selection outperform traditional IE-based methods across diverse datasets? (2) Do all FMs perform equally well as IEs for subset selection? Extensive experiments uncovered surprising insights: FMs consistently outperform traditional IEs on fine-grained datasets, whereas their advantage diminishes on coarse-grained datasets with noisy labels. Motivated by these finding, we propose RAM-APL (RAnking Mean-Accuracy of Pseudo-class Labels), a method tailored for fine-grained image datasets. RAM-APL leverages multiple FMs to enhance subset selection by exploiting their complementary strengths. Our approach achieves state-of-the-art performance on fine-grained datasets, including Oxford-IIIT Pet, Food-101, and Caltech-UCSD Birds-200-2011.