One Head Eight Arms: Block Matrix based Low Rank Adaptation for CLIP-based Few-Shot Learning
This work addresses efficiency challenges in fine-tuning large foundation models for few-shot learning, offering a more practical solution for researchers and practitioners with limited resources.
The paper tackles the problem of excessive training parameters and high computational costs in fine-tuning Vision-Language Models for few-shot learning by proposing Block-LoRA, a block matrix-based low-rank adaptation framework, which reduces parameters and computational costs, enabling fine-tuning on ImageNet with a single 24GB GPU while achieving competitive performance.
Recent advancements in fine-tuning Vision-Language Foundation Models (VLMs) have garnered significant attention for their effectiveness in downstream few-shot learning tasks.While these recent approaches exhibits some performance improvements, they often suffer from excessive training parameters and high computational costs. To address these challenges, we propose a novel Block matrix-based low-rank adaptation framework, called Block-LoRA, for fine-tuning VLMs on downstream few-shot tasks. Inspired by recent work on Low-Rank Adaptation (LoRA), Block-LoRA partitions the original low-rank decomposition matrix of LoRA into a series of sub-matrices while sharing all down-projection sub-matrices. This structure not only reduces the number of training parameters, but also transforms certain complex matrix multiplication operations into simpler matrix addition, significantly lowering the computational cost of fine-tuning. Notably, Block-LoRA enables fine-tuning CLIP on the ImageNet few-shot benchmark using a single 24GB GPU. We also show that Block-LoRA has the more tighter bound of generalization error than vanilla LoRA. Without bells and whistles, extensive experiments demonstrate that Block-LoRA achieves competitive performance compared to state-of-the-art CLIP-based few-shot methods, while maintaining a low training parameters count and reduced computational overhead.