Few-shot Algorithm Assurance
This addresses model reliability for deployment in image classification, but it is incremental as it builds on existing assurance and few-shot learning techniques.
The paper tackles the problem of determining whether a deep learning model's accuracy remains above a threshold under image distortion, proposing a classifier based on Level Set Estimation and extending it to few-shot settings with synthetic image generation. Experiments show the method significantly outperforms baselines on five benchmark datasets.
In image classification tasks, deep learning models are vulnerable to image distortion. For successful deployment, it is important to identify distortion levels under which the model is usable i.e. its accuracy stays above a stipulated threshold. We refer to this problem as Model Assurance under Image Distortion, and formulate it as a classification task. Given a distortion level, our goal is to predict if the model's accuracy on the set of distorted images is greater than a threshold. We propose a novel classifier based on a Level Set Estimation (LSE) algorithm, which uses the LSE's mean and variance functions to form the classification rule. We further extend our method to a "few sample" setting where we can only acquire few real images to perform the model assurance process. Our idea is to generate extra synthetic images using a novel Conditional Variational Autoencoder model with two new loss functions. We conduct extensive experiments to show that our classification method significantly outperforms strong baselines on five benchmark image datasets.