IAA: Inner-Adaptor Architecture Empowers Frozen Large Language Model with Multimodal Capabilities
This addresses the issue of performance degradation in MLLMs for researchers and practitioners, representing a novel method rather than an incremental improvement.
The paper tackles the problem of multimodal large language models (MLLMs) losing natural language processing (NLP) capabilities when fine-tuned with vision-language data by proposing the Inner-Adaptor Architecture (IAA), which freezes the language model and uses multimodal adaptors to achieve superior performance on vision-language benchmarks without sacrificing NLP performance.
In the field of multimodal large language models (MLLMs), common methods typically involve unfreezing the language model during training to foster profound visual understanding. However, the fine-tuning of such models with vision-language data often leads to a diminution of their natural language processing (NLP) capabilities. To avoid this performance degradation, a straightforward solution is to freeze the language model while developing multimodal competencies. Unfortunately, previous works have not attained satisfactory outcomes. Building on the strategy of freezing the language model, we conduct thorough structural exploration and introduce the Inner-Adaptor Architecture (IAA). Specifically, the architecture incorporates multiple multimodal adaptors at varying depths within the large language model to facilitate direct interaction with the inherently text-oriented transformer layers, thereby enabling the frozen language model to acquire multimodal capabilities. Unlike previous approaches of freezing language models that require large-scale aligned data, our proposed architecture is able to achieve superior performance on small-scale datasets. We conduct extensive experiments to improve the general multimodal capabilities and visual grounding abilities of the MLLM. Our approach remarkably outperforms previous state-of-the-art methods across various vision-language benchmarks without sacrificing performance on NLP tasks. Code and models are available at https://github.com/360CVGroup/Inner-Adaptor-Architecture.