Human and AI Perceptual Differences in Image Classification Errors
This addresses the problem of understanding AI-human perceptual gaps for improving collaborative systems, though it is incremental in exploring existing differences.
The study analyzed perceptual differences between humans and AI in image classification errors, finding significant and consistent differences even when AI outperforms humans in accuracy, and demonstrated a human-AI teaming algorithm that surpassed individual or AI-AI teaming performance.
Artificial intelligence (AI) models for computer vision trained with supervised machine learning are assumed to solve classification tasks by imitating human behavior learned from training labels. Most efforts in recent vision research focus on measuring the model task performance using standardized benchmarks such as accuracy. However, limited work has sought to understand the perceptual difference between humans and machines. To fill this gap, this study first analyzes the statistical distributions of mistakes from the two sources and then explores how task difficulty level affects these distributions. We find that even when AI learns an excellent model from the training data, one that outperforms humans in overall accuracy, these AI models have significant and consistent differences from human perception. We demonstrate the importance of studying these differences with a simple human-AI teaming algorithm that outperforms humans alone, AI alone, or AI-AI teaming.