RSIR Transformer: Hierarchical Vision Transformer using Random Sampling Windows and Important Region Windows
This work addresses efficiency and context modeling challenges in vision transformers for high-resolution tasks, presenting an incremental improvement over existing local attention methods.
The paper tackles the problem of high computational cost and limited receptive fields in vision transformers by introducing random sampling windows (RS-Win) and important region windows (IR-Win) to enhance global modeling. It reports competitive performance on common vision tasks, though no specific numerical results are provided in the abstract.
Recently, Transformers have shown promising performance in various vision tasks. However, the high costs of global self-attention remain challenging for Transformers, especially for high-resolution vision tasks. Local self-attention runs attention computation within a limited region for the sake of efficiency, resulting in insufficient context modeling as their receptive fields are small. In this work, we introduce two new attention modules to enhance the global modeling capability of the hierarchical vision transformer, namely, random sampling windows (RS-Win) and important region windows (IR-Win). Specifically, RS-Win sample random image patches to compose the window, following a uniform distribution, i.e., the patches in RS-Win can come from any position in the image. IR-Win composes the window according to the weights of the image patches in the attention map. Notably, RS-Win is able to capture global information throughout the entire model, even in earlier, high-resolution stages. IR-Win enables the self-attention module to focus on important regions of the image and capture more informative features. Incorporated with these designs, RSIR-Win Transformer demonstrates competitive performance on common vision tasks.