Selective Feature Adapter for Dense Vision Transformers
This addresses the storage and computational burden for researchers and practitioners using vision transformers in dense prediction tasks, offering an incremental improvement in parameter efficiency.
The paper tackles the high parameter cost of fine-tuning dense vision transformers by proposing a selective feature adapter (SFA) that achieves state-of-the-art performance under any trainable parameter budget and matches or exceeds fully fine-tuned models across tasks like segmentation, detection, and depth-estimation.
Fine-tuning pre-trained transformer models, e.g., Swin Transformer, are successful in numerous downstream for dense prediction vision tasks. However, one major issue is the cost/storage of their huge amount of parameters, which becomes increasingly challenging to handle with the growing amount of vision tasks. In this paper, we propose an effective approach to alleviate the issue, namely selective feature adapter (SFA). It achieves state-of-the-art (SoTA) performance under any given budget of trainable parameters, and demonstrates comparable or better performance than fully fine-tuned models across various dense tasks. Specifically, SFA consists of external adapters and internal adapters which are sequentially operated over a transformer model. For external adapters, we properly select the places and amount of additional multilayer perception (MLP). For internal adapters, we transform a few task-important parameters inside the transformer, which are automatically discovered through a simple yet effective lottery ticket algorithm. Our experiments show that the dual adapter module, a.k.a SFA, is essential to achieve the best trade-off on dense vision tasks, such as segmentation, detection and depth-estimation, outperforming other adapters with a single module.