GTA: Guided Transfer of Spatial Attention from Object-Centric Representations
This addresses overfitting in ViT for computer vision tasks, offering a domain-specific improvement for transfer learning with small datasets.
The paper tackles the problem of representation deterioration in Vision Transformers (ViT) during transfer learning on small datasets, proposing a regularization method called Guided Transfer of spatial Attention (GTA) that improves accuracy across five benchmark datasets, especially with limited data.
Utilizing well-trained representations in transfer learning often results in superior performance and faster convergence compared to training from scratch. However, even if such good representations are transferred, a model can easily overfit the limited training dataset and lose the valuable properties of the transferred representations. This phenomenon is more severe in ViT due to its low inductive bias. Through experimental analysis using attention maps in ViT, we observe that the rich representations deteriorate when trained on a small dataset. Motivated by this finding, we propose a novel and simple regularization method for ViT called Guided Transfer of spatial Attention (GTA). Our proposed method regularizes the self-attention maps between the source and target models. A target model can fully exploit the knowledge related to object localization properties through this explicit regularization. Our experimental results show that the proposed GTA consistently improves the accuracy across five benchmark datasets especially when the number of training data is small.