CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning
This work solves the problem of balancing forgetting and parameter efficiency for MLLMs in continual learning, which is critical for real-world deployment.
CRAM addresses the dilemma between catastrophic forgetting and parameter inefficiency in multimodal continual instruction tuning by isolating task-specific patterns into independent modules and dynamically allocating parameters based on capability gaps, achieving superior performance across diverse benchmarks.
Multimodal Large Language Models (MLLMs) unify heterogeneous vision-language tasks under a shared generative framework via instruction tuning, yet real-world deployment demands continuous capability expansion, making Multimodal Continual Instruction Tuning (MCIT) essential. Existing methods either update all tasks with a shared parameter set or allocate dedicated modules for each new task. Shared updates force heterogeneous tasks to compete, causing forgetting of learned capabilities. Conversely, isolated expansion prevents interference but severely limits parameter efficiency over long task streams. To address this dilemma, we propose CRAM. Specifically, by isolating task-specific patterns into independent modules, CRAM mitigates catastrophic forgetting across tasks. To further boost parameter efficiency, we utilize adaptive-rank instantiation to identify the capability gap between existing expert capability and new task demands, and dynamically allocate only the necessary parameters. To ensure stable reuse among tasks, centroid-guided routing recognizes and activates existing experts' capabilities, while an orthogonality penalty confines new updates to task-specific directions, preventing re-learning general capability. Extensive experiments across diverse benchmarks consistently demonstrate its superiority over existing methods.