Revisiting Fine-tuning for Few-shot Learning
This work addresses the challenge of learning novel classes with few examples for machine learning practitioners, but it is incremental as it re-evaluates an existing method rather than introducing a new paradigm.
The study tackled the problem of few-shot learning by revisiting fine-tuning, showing that it achieves higher accuracy than common few-shot learning algorithms on low-resolution mini-ImageNet (e.g., in 1-shot tasks) and comparable accuracy in 5-shot tasks, with similar gains in high-resolution single-domain and cross-domain tasks.
Few-shot learning is the process of learning novel classes using only a few examples and it remains a challenging task in machine learning. Many sophisticated few-shot learning algorithms have been proposed based on the notion that networks can easily overfit to novel examples if they are simply fine-tuned using only a few examples. In this study, we show that in the commonly used low-resolution mini-ImageNet dataset, the fine-tuning method achieves higher accuracy than common few-shot learning algorithms in the 1-shot task and nearly the same accuracy as that of the state-of-the-art algorithm in the 5-shot task. We then evaluate our method with more practical tasks, namely the high-resolution single-domain and cross-domain tasks. With both tasks, we show that our method achieves higher accuracy than common few-shot learning algorithms. We further analyze the experimental results and show that: 1) the retraining process can be stabilized by employing a low learning rate, 2) using adaptive gradient optimizers during fine-tuning can increase test accuracy, and 3) test accuracy can be improved by updating the entire network when a large domain-shift exists between base and novel classes.