CROME: Cross-Modal Adapters for Efficient Multimodal LLM
This work addresses the problem of expensive training and limited adaptability for multimodal models, offering a more efficient solution for researchers and practitioners, though it is incremental as it builds on existing adapter-based methods.
The paper tackles the challenge of cost-effective training and adaptation for multimodal large language models (MLLMs) by proposing CROME, a framework with a gated cross-modal adapter that achieves superior zero-shot performance on visual question answering and instruction-following benchmarks and competes with state-of-the-art methods in fine-tuning with high parameter efficiency.
Multimodal Large Language Models (MLLMs) demonstrate remarkable image-language capabilities, but their widespread use faces challenges in cost-effective training and adaptation. Existing approaches often necessitate expensive language model retraining and limited adaptability. Additionally, the current focus on zero-shot performance improvements offers insufficient guidance for task-specific tuning. We propose CROME, an efficient vision-language instruction tuning framework. It features a novel gated cross-modal adapter that effectively combines visual and textual representations prior to input into a frozen LLM. This lightweight adapter, trained with minimal parameters, enables efficient cross-modal understanding. Notably, CROME demonstrates superior zero-shot performance on standard visual question answering and instruction-following benchmarks. Moreover, it yields fine-tuning with exceptional parameter efficiency, competing with task-specific specialist state-of-the-art methods. CROME demonstrates the potential of pre-LM alignment for building scalable, adaptable, and parameter-efficient multimodal models.