Enhancing Multi-modal Models with Heterogeneous MoE Adapters for Fine-tuning
This addresses the problem of efficient fine-tuning for multi-modal tasks, offering a domain-specific improvement over existing uni-modal methods.
The paper tackles the computational expense of multi-modal models by proposing heterogeneous mixture of experts adapters for parameter-efficient fine-tuning, achieving competitive performance with only 5-8% of parameters fine-tuned across eight downstream tasks.
Multi-modal models excel in cross-modal tasks but are computationally expensive due to their billions of parameters. Parameter-efficient fine-tuning (PEFT) offers a solution by adding small trainable components while freezing pre-trained parameters. However, existing methods primarily focus on uni-modal processing, overlooking the critical modal fusion needed for multi-modal tasks. To fill this gap, we propose heterogeneous mixture of experts adapters that extend the traditional PEFT framework to support multi-modal expert combinations and improve information interaction. Additionally, our approach modifies the affine linear expert design to enable efficient modal fusion in a low-rank space, achieving competitive performance with only 5-8\% of the parameters fine-tuned. Experiments across eight downstream tasks, including visual-audio and text-visual, demonstrate the superior performance of the approach.