Selective State Space Memory for Large Vision-Language Models
This addresses the problem of efficient fine-tuning for large vision-language models, which is incremental as it builds on existing methods with a novel integration approach.
The paper tackles the computationally intensive challenge of fine-tuning large vision-language models for domain-specific applications by introducing State Space Memory Integration (SSMI), which achieves state-of-the-art performance on benchmark datasets like COCO Captioning, VQA, and Flickr30k while requiring only a fraction of the model's parameters to be updated.
Large Vision-Language Models (LVLMs) have demonstrated remarkable performance across a wide range of multimodal tasks. However, fine-tuning these models for domain-specific applications remains a computationally intensive challenge. This paper introduces State Space Memory Integration (SSMI), a novel approach for efficient fine-tuning of LVLMs. By integrating lightweight Mamba-based state space modules into the LVLM architecture, SSMI captures long-range dependencies and injects task-specific visual and sequential patterns effectively. Unlike traditional fine-tuning methods, SSMI requires only a fraction of the model's parameters to be updated, making it computationally efficient and scalable. Experiments on benchmark datasets, including COCO Captioning, VQA, and Flickr30k, demonstrate that SSMI achieves state-of-the-art performance while maintaining robustness and generalization capabilities. Comprehensive analysis further validates the advantages of SSMI in terms of efficiency, adaptability, and interpretability, positioning it as a compelling solution for fine-tuning large-scale vision-language models.