Predicting the Reliability of an Image Classifier under Image Distortion
This addresses quality control for image classification systems by predicting reliability under distortions, but it is incremental as it focuses on handling imbalanced training data.
The paper tackles the problem of predicting whether an image classifier remains reliable under image distortions by training a distortion-classifier, and it shows that their Gaussian process-based method significantly outperforms baselines on six datasets.
In image classification tasks, deep learning models are vulnerable to image distortions i.e. their accuracy significantly drops if the input images are distorted. An image-classifier is considered "reliable" if its accuracy on distorted images is above a user-specified threshold. For a quality control purpose, it is important to predict if the image-classifier is unreliable/reliable under a distortion level. In other words, we want to predict whether a distortion level makes the image-classifier "non-reliable" or "reliable". Our solution is to construct a training set consisting of distortion levels along with their "non-reliable" or "reliable" labels, and train a machine learning predictive model (called distortion-classifier) to classify unseen distortion levels. However, learning an effective distortion-classifier is a challenging problem as the training set is highly imbalanced. To address this problem, we propose a Gaussian process based method to rebalance the training set. We conduct extensive experiments to show that our method significantly outperforms several baselines on six popular image datasets.