Leveraging Pre-Trained Models for Multimodal Class-Incremental Learning under Adaptive Fusion
This addresses the problem of catastrophic forgetting in multimodal AI systems for applications like robotics or smart assistants, though it is incremental as it builds on existing MCIL frameworks.
The paper tackles multimodal class-incremental learning across vision, audio, and text by proposing a method based on pre-trained models, achieving state-of-the-art performance with improvements of 2.1-4.5% in average accuracy over existing methods on three datasets.
Unlike traditional Multimodal Class-Incremental Learning (MCIL) methods that focus only on vision and text, this paper explores MCIL across vision, audio and text modalities, addressing challenges in integrating complementary information and mitigating catastrophic forgetting. To tackle these issues, we propose an MCIL method based on multimodal pre-trained models. Firstly, a Multimodal Incremental Feature Extractor (MIFE) based on Mixture-of-Experts (MoE) structure is introduced to achieve effective incremental fine-tuning for AudioCLIP. Secondly, to enhance feature discriminability and generalization, we propose an Adaptive Audio-Visual Fusion Module (AAVFM) that includes a masking threshold mechanism and a dynamic feature fusion mechanism, along with a strategy to enhance text diversity. Thirdly, a novel multimodal class-incremental contrastive training loss is proposed to optimize cross-modal alignment in MCIL. Finally, two MCIL-specific evaluation metrics are introduced for comprehensive assessment. Extensive experiments on three multimodal datasets validate the effectiveness of our method.