1%>100%: High-Efficiency Visual Adapter with Complex Linear Projection Optimization
This provides an efficient solution for deploying vision foundation models, addressing high costs and low efficiency in adaptation, though it is incremental as it builds on delta-tuning concepts.
The paper tackles the inefficiency of adapting vision foundation models by proposing CoLin, a low-rank complex adapter that introduces only about 1% parameters and outperforms full fine-tuning and delta-tuning methods across multiple vision tasks.
Deploying vision foundation models typically relies on efficient adaptation strategies, whereas conventional full fine-tuning suffers from prohibitive costs and low efficiency. While delta-tuning has proven effective in boosting the performance and efficiency of LLMs during adaptation, its advantages cannot be directly transferred to the fine-tuning pipeline of vision foundation models. To push the boundaries of adaptation efficiency for vision tasks, we propose an adapter with Complex Linear Projection Optimization (CoLin). For architecture, we design a novel low-rank complex adapter that introduces only about 1% parameters to the backbone. For efficiency, we theoretically prove that low-rank composite matrices suffer from severe convergence issues during training, and address this challenge with a tailored loss. Extensive experiments on object detection, segmentation, image classification, and rotated object detection (remote sensing scenario) demonstrate that CoLin outperforms both full fine-tuning and classical delta-tuning approaches with merely 1% parameters for the first time, providing a novel and efficient solution for deployment of vision foundation models. We release the code on https://github.com/DongshuoYin/CoLin.