Comparing object recognition in humans and deep convolutional neural networks -- An eye tracking study
This proof-of-concept study addresses the problem of comparing visual processing in humans and AI for researchers in biological and computer vision, though it is incremental in nature.
The study compared object recognition in humans and deep convolutional neural networks (DCNNs) using eye tracking and saliency maps, finding that a biologically plausible DCNN (vNet) showed higher agreement with human viewing behavior than a standard ResNet, with image-specific factors like category and animacy affecting this agreement.
Deep convolutional neural networks (DCNNs) and the ventral visual pathway share vast architectural and functional similarities in visual challenges such as object recognition. Recent insights have demonstrated that both hierarchical cascades can be compared in terms of both exerted behavior and underlying activation. However, these approaches ignore key differences in spatial priorities of information processing. In this proof-of-concept study, we demonstrate a comparison of human observers (N = 45) and three feedforward DCNNs through eye tracking and saliency maps. The results reveal fundamentally different resolutions in both visualization methods that need to be considered for an insightful comparison. Moreover, we provide evidence that a DCNN with biologically plausible receptive field sizes called vNet reveals higher agreement with human viewing behavior as contrasted with a standard ResNet architecture. We find that image-specific factors such as category, animacy, arousal, and valence have a direct link to the agreement of spatial object recognition priorities in humans and DCNNs, while other measures such as difficulty and general image properties do not. With this approach, we try to open up new perspectives at the intersection of biological and computer vision research.