MCITlib: Multimodal Continual Instruction Tuning Library and Benchmark
This work provides a tool for researchers in multimodal continual learning, but it is incremental as it builds on existing methods by extending them to multimodal settings.
The paper introduces MCITlib, a library and benchmark for multimodal continual instruction tuning, implementing 8 algorithms and evaluating them on 2 benchmarks to address catastrophic forgetting and cross-modal challenges in multimodal large language models.
Continual learning aims to equip AI systems with the ability to continuously acquire and adapt to new knowledge without forgetting previously learned information, similar to human learning. While traditional continual learning methods focusing on unimodal tasks have achieved notable success, the emergence of Multimodal Large Language Models has brought increasing attention to Multimodal Continual Learning tasks involving multiple modalities, such as vision and language. In this setting, models are expected to not only mitigate catastrophic forgetting but also handle the challenges posed by cross-modal interactions and coordination. To facilitate research in this direction, we introduce MCITlib, a comprehensive and constantly evolving code library for continual instruction tuning of Multimodal Large Language Models. In MCITlib, we have currently implemented 8 representative algorithms for Multimodal Continual Instruction Tuning and systematically evaluated them on 2 carefully selected benchmarks. MCITlib will be continuously updated to reflect advances in the Multimodal Continual Learning field. The codebase is released at https://github.com/Ghy0501/MCITlib.