APLA: A Simple Adaptation Method for Vision Transformers
This addresses the need for simpler and more efficient adaptation methods for vision transformers in various domains like medical and satellite imaging, though it is incremental as it builds on existing adaptation techniques.
The paper tackles the problem of adapting vision transformers efficiently by introducing APLA, a method that updates only the projection layer after attention, achieving state-of-the-art performance across 46 datasets while reducing GPU memory usage by up to 52.63% and training time by up to 43.0%.
Existing adaptation techniques typically require architectural modifications or added parameters, leading to high computational costs and complexity. We introduce Attention Projection Layer Adaptation (APLA), a simple approach to adapt vision transformers (ViTs) without altering the architecture or adding parameters. Through a systematic analysis, we find that the layer immediately after the attention mechanism is crucial for adaptation. By updating only this projection layer, or even just a random subset of this layer's weights, APLA achieves state-of-the-art performance while reducing GPU memory usage by up to 52.63% and training time by up to 43.0%, with no extra cost at inference. Across 46 datasets covering a variety of tasks including scene classification, medical imaging, satellite imaging, and fine-grained classification, APLA consistently outperforms 17 other leading adaptation methods, including full fine-tuning, on classification, segmentation, and detection tasks. The code is available at https://github.com/MoeinSorkhei/APLA.