Fine-tuning MLLMs Without Forgetting Is Easier Than You Think
This work provides practical guidelines for adapting MLLMs without forgetting, addressing a key problem for researchers and practitioners in multimodal AI, though it is incremental as it builds on existing fine-tuning methods.
The paper tackled catastrophic forgetting in fine-tuning multimodal large language models (MLLMs) by showing that simple adjustments like regularization and low learning rates mitigate forgetting for out-of-distribution images, but identified task-specific overfitting for in-distribution images with out-of-distribution text, which they addressed with a data-hybrid training strategy that outperforms existing methods in continual learning.
The paper demonstrate that simple adjustments of the fine-tuning recipes of multimodal large language models (MLLM) are sufficient to mitigate catastrophic forgetting. On visual question answering, we design a 2x2 experimental framework to assess model performance across in-distribution and out-of-distribution image and text inputs. Our results show that appropriate regularization, such as constraining the number of trainable parameters or adopting a low learning rate, effectively prevents forgetting when dealing with out-of-distribution images. However, we uncover a distinct form of forgetting in settings with in-distribution images and out-of-distribution text. We attribute this forgetting as task-specific overfitting and address this issue by introducing a data-hybrid training strategy that combines datasets and tasks. Finally, we demonstrate that this approach naturally extends to continual learning, outperforming existing methods with complex auxiliary mechanisms. In general, our findings challenge the prevailing assumptions by highlighting the inherent robustness of MLLMs and providing practical guidelines for adapting them while preserving their general capabilities.