Prior Knowledge-Guided Attention in Self-Supervised Vision Transformers
This addresses data efficiency and interpretability issues in medical image analysis and visual quality assurance, representing an incremental improvement over existing self-supervised methods.
The paper tackles the problem of limited data availability in self-supervised vision transformers by introducing spatial prior attention (SPAN), which guides attention using spatial and semantic priors derived from data statistics or minimal labeled samples. The result includes a 58.7 mAP improvement for lung and heart segmentation and a 2.2 mAUC gain in chest disease classification compared to domain-agnostic pretraining.
Recent trends in self-supervised representation learning have focused on removing inductive biases from training pipelines. However, inductive biases can be useful in settings when limited data are available or provide additional insight into the underlying data distribution. We present spatial prior attention (SPAN), a framework that takes advantage of consistent spatial and semantic structure in unlabeled image datasets to guide Vision Transformer attention. SPAN operates by regularizing attention masks from separate transformer heads to follow various priors over semantic regions. These priors can be derived from data statistics or a single labeled sample provided by a domain expert. We study SPAN through several detailed real-world scenarios, including medical image analysis and visual quality assurance. We find that the resulting attention masks are more interpretable than those derived from domain-agnostic pretraining. SPAN produces a 58.7 mAP improvement for lung and heart segmentation. We also find that our method yields a 2.2 mAUC improvement compared to domain-agnostic pretraining when transferring the pretrained model to a downstream chest disease classification task. Lastly, we show that SPAN pretraining leads to higher downstream classification performance in low-data regimes compared to domain-agnostic pretraining.