Automatic Combination of Sample Selection Strategies for Few-Shot Learning
This work addresses the problem of optimizing sample selection for improved few-shot learning performance, which is incremental as it builds on existing strategies by combining them.
The paper investigated the impact of 20 sample selection strategies on 5 few-shot learning approaches across 14 datasets, finding strong dependencies on modality, dataset, and shot count, and proposed ACSESS, a method that automatically combines strategies to outperform individual ones and a recent baseline.
In few-shot learning, such as meta-learning, few-shot fine-tuning or in-context learning, the limited number of samples used to train a model have a significant impact on the overall success. Although a large number of sample selection strategies exist, their impact on the performance of few-shot learning is not extensively known, as most of them have been so far evaluated in typical supervised settings only. In this paper, we thoroughly investigate the impact of 20 sample selection strategies on the performance of 5 few-shot learning approaches over 8 image and 6 text datasets. In addition, we propose a new method for automatic combination of sample selection strategies (ACSESS) that leverages the strengths and complementary information of the individual strategies. The experimental results show that our method consistently outperforms the individual selection strategies, as well as the recently proposed method for selecting support examples for in-context learning. We also show a strong modality, dataset and approach dependence for the majority of strategies as well as their dependence on the number of shots - demonstrating that the sample selection strategies play a significant role for lower number of shots, but regresses to random selection at higher number of shots.