Learn from Downstream and Be Yourself in Multimodal Large Language Model Fine-Tuning
This addresses a key trade-off between specialization and generalization in fine-tuning multimodal LLMs, though it appears incremental as it builds on existing parameter importance methods.
The paper tackles the problem of multimodal large language models forgetting pre-trained knowledge during fine-tuning, which reduces generalization. They propose measuring parameter importance from both pre-training and fine-tuning data, then selectively updating important parameters, achieving improved downstream performance while mitigating generalization degradation in image captioning and VQA tasks.
Multimodal Large Language Model (MLLM) have demonstrated strong generalization capabilities across diverse distributions and tasks, largely due to extensive pre-training datasets. Fine-tuning MLLM has become a common practice to improve performance on specific downstream tasks. However, during fine-tuning, MLLM often faces the risk of forgetting knowledge acquired during pre-training, which can result in a decline in generalization abilities. To balance the trade-off between generalization and specialization, we propose measuring the parameter importance for both pre-trained and fine-tuning distributions, based on frozen pre-trained weight magnitude and accumulated fine-tuning gradient values. We further apply an importance-aware weight allocation strategy, selectively updating relatively important parameters for downstream tasks. We conduct empirical evaluations on both image captioning and visual question-answering tasks using various MLLM architectures. The comprehensive experimental analysis demonstrates the effectiveness of the proposed solution, highlighting the efficiency of the crucial modules in enhancing downstream specialization performance while mitigating generalization degradation in MLLM Fine-Tuning.