IBiT: Utilizing Inductive Biases to Create a More Data Efficient Attention Mechanism
This work addresses the data inefficiency of Vision Transformers for computer vision applications, offering a more practical solution for scenarios with limited data.
The paper tackled the problem of Vision Transformers requiring large datasets by introducing inductive biases from CNNs through learned masks, resulting in IBiT achieving significantly higher accuracy on small datasets without needing knowledge distillation.
In recent years, Transformer-based architectures have become the dominant method for Computer Vision applications. While Transformers are explainable and scale well with dataset size, they lack the inductive biases of Convolutional Neural Networks. While these biases may be learned on large datasets, we show that introducing these inductive biases through learned masks allow Vision Transformers to learn on much smaller datasets without Knowledge Distillation. These Transformers, which we call Inductively Biased Image Transformers (IBiT), are significantly more accurate on small datasets, while retaining the explainability Transformers.