Passive Attention in Artificial Neural Networks Predicts Human Visual Selectivity
This work provides a new method to assess the biological validity of ANNs as models of human vision by comparing visual selectivity, though it is incremental in applying existing interpretability techniques to human data.
The study investigated whether artificial neural networks (ANNs) and humans prioritize similar image regions for visual tasks, finding that passive attention techniques in ANNs significantly overlap with human visual selectivity across multiple behavioral measures, with ANN attention maps improving human classification speed and human maps affecting ANN recognition performance.
Developments in machine learning interpretability techniques over the past decade have provided new tools to observe the image regions that are most informative for classification and localization in artificial neural networks (ANNs). Are the same regions similarly informative to human observers? Using data from 79 new experiments and 7,810 participants, we show that passive attention techniques reveal a significant overlap with human visual selectivity estimates derived from 6 distinct behavioral tasks including visual discrimination, spatial localization, recognizability, free-viewing, cued-object search, and saliency search fixations. We find that input visualizations derived from relatively simple ANN architectures probed using guided backpropagation methods are the best predictors of a shared component in the joint variability of the human measures. We validate these correlational results with causal manipulations using recognition experiments. We show that images masked with ANN attention maps were easier for humans to classify than control masks in a speeded recognition experiment. Similarly, we find that recognition performance in the same ANN models was likewise influenced by masking input images using human visual selectivity maps. This work contributes a new approach to evaluating the biological and psychological validity of leading ANNs as models of human vision: by examining their similarities and differences in terms of their visual selectivity to the information contained in images.