Tune It or Don't Use It: Benchmarking Data-Efficient Image Classification
This work addresses the need for fair comparisons in data-efficient image classification, revealing that incremental tuning often suffices over specialized methods.
The authors tackled the problem of inconsistent benchmarking in data-efficient image classification by creating a diverse six-dataset benchmark and re-evaluating eight methods with careful hyperparameter tuning. They found that a tuned baseline outperformed all but one specialized method and was competitive with the remaining one.
Data-efficient image classification using deep neural networks in settings, where only small amounts of labeled data are available, has been an active research area in the recent past. However, an objective comparison between published methods is difficult, since existing works use different datasets for evaluation and often compare against untuned baselines with default hyper-parameters. We design a benchmark for data-efficient image classification consisting of six diverse datasets spanning various domains (e.g., natural images, medical imagery, satellite data) and data types (RGB, grayscale, multispectral). Using this benchmark, we re-evaluate the standard cross-entropy baseline and eight methods for data-efficient deep learning published between 2017 and 2021 at renowned venues. For a fair and realistic comparison, we carefully tune the hyper-parameters of all methods on each dataset. Surprisingly, we find that tuning learning rate, weight decay, and batch size on a separate validation split results in a highly competitive baseline, which outperforms all but one specialized method and performs competitively to the remaining one.