Efficient Image Dataset Classification Difficulty Estimation for Predicting Deep-Learning Accuracy
This addresses the challenge for researchers and practitioners in quickly evaluating algorithms without expensive searches, though it is incremental as it builds on existing optimization methods.
The paper tackles the problem of efficiently selecting deep-learning algorithms for new image classification datasets by proposing a method to estimate dataset classification difficulty, which computes a single number 27x faster than training state-of-the-art networks.
In the deep-learning community new algorithms are published at an incredible pace. Therefore, solving an image classification problem for new datasets becomes a challenging task, as it requires to re-evaluate published algorithms and their different configurations in order to find a close to optimal classifier. To facilitate this process, before biasing our decision towards a class of neural networks or running an expensive search over the network space, we propose to estimate the classification difficulty of the dataset. Our method computes a single number that characterizes the dataset difficulty 27x faster than training state-of-the-art networks. The proposed method can be used in combination with network topology and hyper-parameter search optimizers to efficiently drive the search towards promising neural-network configurations.