Affinity-based Attention in Self-supervised Transformers Predicts Dynamics of Object Grouping in Humans
This work addresses the challenge of understanding visual grouping mechanisms in humans, providing new benchmarks for evaluating visual representation learning models, though it is incremental as it builds on existing self-supervised Transformer methods.
The study tackled the problem of modeling human object-based attention and segmentation in natural images by proposing an affinity-based attention spreading mechanism using self-supervised vision Transformers, resulting in significant improvement over baseline and CNN models in predicting human reaction time patterns, with concrete performance gains reported.
The spreading of attention has been proposed as a mechanism for how humans group features to segment objects. However, such a mechanism has not yet been implemented and tested in naturalistic images. Here, we leverage the feature maps from self-supervised vision Transformers and propose a model of human object-based attention spreading and segmentation. Attention spreads within an object through the feature affinity signal between different patches of the image. We also collected behavioral data on people grouping objects in natural images by judging whether two dots are on the same object or on two different objects. We found that our models of affinity spread that were built on feature maps from the self-supervised Transformers showed significant improvement over baseline and CNN based models on predicting reaction time patterns of humans, despite not being trained on the task or with any other object labels. Our work provides new benchmarks for evaluating models of visual representation learning including Transformers.