Towards Minimal Fine-Tuning of VLMs
This work addresses the need for parameter-efficient fine-tuning in vision-language models, offering an incremental improvement for researchers and practitioners in computer vision and NLP.
The paper tackles the problem of fine-tuning vision-language models efficiently by introducing Image-LoRA, a lightweight method that reduces adapter-only training FLOPs in proportion to the visual-token fraction, achieving comparable accuracy to standard LoRA with fewer parameters and lower FLOPs across various benchmarks.
We introduce Image-LoRA, a lightweight parameter efficient fine-tuning (PEFT) recipe for transformer-based vision-language models (VLMs). Image-LoRA applies low-rank adaptation only to the value path of attention layers within the visual-token span, reducing adapter-only training FLOPs roughly in proportion to the visual-token fraction. We further adapt only a subset of attention heads, selected using head influence scores estimated with a rank-1 Image-LoRA, and stabilize per-layer updates via selection-size normalization. Across screen-centric grounding and referring benchmarks spanning text-heavy to image-heavy regimes, Image-LoRA matches or closely approaches standard LoRA accuracy while using fewer trainable parameters and lower adapter-only training FLOPs. The method also preserves the pure-text reasoning performance of VLMs before and after fine-tuning, as further shown on GSM8K.