Novel Deep Neural Network Classifier Characterization Metrics with Applications to Dataless Evaluation
This addresses the challenge for users with limited test data in selecting among many open-source classifiers, though it is incremental as it builds on existing DNN architectures.
The paper tackled the problem of evaluating deep neural network classifiers without using test data by proposing metrics based on weight vectors and synthetic inputs, and empirically confirmed its feasibility on ResNet18 trained with CIFAR10 and CIFAR100 datasets.
The mainstream AI community has seen a rise in large-scale open-source classifiers, often pre-trained on vast datasets and tested on standard benchmarks; however, users facing diverse needs and limited, expensive test data may be overwhelmed by available choices. Deep Neural Network (DNN) classifiers undergo training, validation, and testing phases using example dataset, with the testing phase focused on determining the classification accuracy of test examples without delving into the inner working of the classifier. In this work, we evaluate a DNN classifier's training quality without any example dataset. It is assumed that a DNN is a composition of a feature extractor and a classifier which is the penultimate completely connected layer. The quality of a classifier is estimated using its weight vectors. The feature extractor is characterized using two metrics that utilize feature vectors it produces when synthetic data is fed as input. These synthetic input vectors are produced by backpropagating desired outputs of the classifier. Our empirical study of the proposed method for ResNet18, trained with CAFIR10 and CAFIR100 datasets, confirms that data-less evaluation of DNN classifiers is indeed possible.