neuralRank: Searching and ranking ANN-based model repositories
This addresses a critical but overlooked problem for practitioners deploying deep learning models, though it is incremental as it builds on existing model repositories and transfer learning concepts.
The paper tackles the problem of selecting the most suitable pre-trained neural network model from a repository for a given application, proposing the neuralRank algorithm that ranks models based on a novel discriminating power metric. Experimental results on MNIST, Fashion, and CIFAR10 datasets show that neuralRank is domain-independent and highly ranked models tend to have higher accuracy.
Widespread applications of deep learning have led to a plethora of pre-trained neural network models for common tasks. Such models are often adapted from other models via transfer learning. The models may have varying training sets, training algorithms, network architectures, and hyper-parameters. For a given application, what isthe most suitable model in a model repository? This is a critical question for practical deployments but it has not received much attention. This paper introduces the novel problem of searching and ranking models based on suitability relative to a target dataset and proposes a ranking algorithm called \textit{neuralRank}. The key idea behind this algorithm is to base model suitability on the discriminating power of a model, using a novel metric to measure it. With experimental results on the MNIST, Fashion, and CIFAR10 datasets, we demonstrate that (1) neuralRank is independent of the domain, the training set, or the network architecture and (2) that the models ranked highly by neuralRank ranking tend to have higher model accuracy in practice.