Curvature-Guided Mixing for MLLM Adaptation
For practitioners fine-tuning MLLMs on specialized tasks, CGM provides a principled and effective method to mitigate catastrophic forgetting, improving the balance between task performance and general capabilities.
Curvature-Guided Mixing (CGM) addresses catastrophic forgetting in MLLM fine-tuning by merging pre-trained and fine-tuned models using a theoretically grounded, curvature-aware soft mixing ratio derived from Hessian approximations. Experiments on LLaVA-1.5 and Qwen2.5VL show consistent improvements in the trade-off between task specialization and general knowledge retention over existing methods.
Fine-tuning Multimodal Large Language Models (MLLMs) on specialized tasks often leads to catastrophic forgetting of their general capabilities. Existing model merging methods to combat this are often heuristic or use sub-optimal objectives. We propose CurvatureGuided Mixing (CGM), a theoretically grounded framework that merges pre-trained and fine-tuned models. CGM formulates a joint optimization objective and uses a second-order (Hessian) approximation of the loss landscapes to analytically derive an optimal, closed-form "soft mixing" ratio. This ratio intelligently blends parameters based on their relative task-specific curvatures. We also introduce CGM$\dagger$, a robust "hard mixing" variant that performs sparse parameter selection guided by a novel, curvature-aware score. Experiments on LLaVA-1.5 and Qwen2.5VL across multiple downstream tasks show that CGM and CGM$\dagger$ consistently improve the trade-off between task specialization and general knowledge retention over existing methods. Code is available at github.com/zzsyjl/CGM-ECCV-2026.