Harmonious Parameter Adaptation in Continual Visual Instruction Tuning for Safety-Aligned MLLMs
This addresses a critical oversight in adapting real-world MLLMs that require safety mechanisms, though it is incremental as it builds on existing CVIT frameworks.
The paper tackles the problem of continual visual instruction tuning for safety-aligned multimodal large language models, where existing methods cause safety degradation and task forgetting, and proposes Harmonious Parameter Adaptation to better maintain high safety and mitigate forgetting compared to baselines.
While continual visual instruction tuning (CVIT) has shown promise in adapting multimodal large language models (MLLMs), existing studies predominantly focus on models without safety alignment. This critical oversight ignores the fact that real-world MLLMs inherently require such mechanisms to mitigate potential risks. In this work, we shift our focus to CVIT for safety-aligned MLLMs and observe that during continual adaptation, the model not only suffers from task forgetting but also exhibits degradation in its safety. Achieving a harmonious balance between safety and task performance remains a crucial challenge. To address this, we propose Harmonious Parameter Adaptation (HPA), a post-training framework composed of focusing-based parameter partition, harmoniously balanced parameter selection, and orthogonal parameter adjustment. Specifically, HPA partitions parameters into two types based on their focus on safety or task performance, and selects the focused ones to preserve from a balanced perspective. In addition, HPA imposes orthogonality constraints on parameter updates to further alleviate catastrophic forgetting. Extensive experiments on the CVIT benchmark and safety evaluation datasets demonstrate that HPA better maintains high safety and mitigates forgetting than existing baselines.