PCNN: Probable-Class Nearest-Neighbor Explanations Improve Fine-Grained Image Classification Accuracy for AIs and Humans
This work addresses fine-grained image classification for AI systems and human users, offering an incremental improvement by leveraging nearest neighbors to enhance both machine and human performance.
The authors tackled the problem of improving fine-grained image classification accuracy by using nearest neighbors from probable classes to refine a frozen classifier's predictions, achieving consistent accuracy gains on CUB-200, Cars-196, and Dogs-120 datasets, and also found that showing these neighbors to users reduces over-reliance on AI and improves human decision accuracy.
Nearest neighbors (NN) are traditionally used to compute final decisions, e.g., in Support Vector Machines or k-NN classifiers, and to provide users with explanations for the model's decision. In this paper, we show a novel utility of nearest neighbors: To improve predictions of a frozen, pretrained image classifier C. We leverage an image comparator S that (1) compares the input image with NN images from the top-K most probable classes given by C; and (2) uses scores from S to weight the confidence scores of C to refine predictions. Our method consistently improves fine-grained image classification accuracy on CUB-200, Cars-196, and Dogs-120. Also, a human study finds that showing users our probable-class nearest neighbors (PCNN) reduces over-reliance on AI, thus improving their decision accuracy over prior work which only shows only the most-probable (top-1) class examples.