Keep It SimPool: Who Said Supervised Transformers Suffer from Attention Deficit?
This work addresses a bottleneck in supervised vision transformers by offering a universal pooling method that enhances attention quality without architectural changes, which is incremental but impactful for computer vision applications.
The authors tackled the problem of low-quality spatial attention in supervised vision transformers by proposing SimPool, a simple attention-based pooling mechanism that improves performance on pre-training and downstream tasks and provides high-quality attention maps delineating object boundaries in both supervised and self-supervised settings.
Convolutional networks and vision transformers have different forms of pairwise interactions, pooling across layers and pooling at the end of the network. Does the latter really need to be different? As a by-product of pooling, vision transformers provide spatial attention for free, but this is most often of low quality unless self-supervised, which is not well studied. Is supervision really the problem? In this work, we develop a generic pooling framework and then we formulate a number of existing methods as instantiations. By discussing the properties of each group of methods, we derive SimPool, a simple attention-based pooling mechanism as a replacement of the default one for both convolutional and transformer encoders. We find that, whether supervised or self-supervised, this improves performance on pre-training and downstream tasks and provides attention maps delineating object boundaries in all cases. One could thus call SimPool universal. To our knowledge, we are the first to obtain attention maps in supervised transformers of at least as good quality as self-supervised, without explicit losses or modifying the architecture. Code at: https://github.com/billpsomas/simpool.