Towards Efficient Visual-Language Alignment of the Q-Former for Visual Reasoning Tasks
This work addresses the need for more efficient training of multimodal models for researchers and practitioners, but it is incremental as it builds on existing methods like InstructBLIP and PEFT.
The paper tackles the problem of efficiently fine-tuning the Q-Former for visual reasoning tasks by applying parameter efficient fine-tuning (PEFT) methods, achieving comparable performance to full fine-tuning with under 2% of trainable parameters on benchmarks like ScienceQA and IconQA.
Recent advancements in large language models have demonstrated enhanced capabilities in visual reasoning tasks by employing additional encoders for aligning different modalities. While the Q-Former has been widely used as a general encoder for aligning several modalities including image, video, audio, and 3D with large language models, previous works on its efficient training and the analysis of its individual components have been limited. In this work, we investigate the effectiveness of parameter efficient fine-tuning (PEFT) the Q-Former using InstructBLIP with visual reasoning benchmarks ScienceQA and IconQA. We observe that applying PEFT to the Q-Former achieves comparable performance to full fine-tuning using under 2% of the trainable parameters. Additionally, we employ AdaLoRA for dynamic parameter budget reallocation to examine the relative importance of the Q-Former's sublayers with 4 different benchmarks. Our findings reveal that the self-attention layers are noticeably more important in perceptual visual-language reasoning tasks, and relative importance of FFN layers depends on the complexity of visual-language patterns involved in tasks. The code is available at https://github.com/AttentionX/InstructBLIP_PEFT.