FedMLLM: Federated Fine-tuning MLLM on Multimodal Heterogeneity Data
This work addresses privacy-sensitive domains by enabling federated learning for MLLMs, but it is incremental as it builds on existing FL methods with modality-agnostic strategies.
The authors tackled the challenge of multimodal heterogeneities in federated fine-tuning of MLLMs by introducing a benchmark and a framework, which improved MLLM performance by broadening training data and mitigating heterogeneity, though no specific numerical gains were reported.
Multimodal Large Language Models (MLLMs) have made significant advancements, demonstrating powerful capabilities in processing and understanding multimodal data. Fine-tuning MLLMs with Federated Learning (FL) allows for expanding the training data scope by including private data sources, thereby enhancing their practical applicability in privacy-sensitive domains. However, current research remains in the early stage, particularly in addressing the \textbf{multimodal heterogeneities} in real-world applications. In this paper, we introduce a benchmark to evaluate the performance of federated fine-tuning of MLLMs across various multimodal heterogeneous scenarios, laying the groundwork for future research in the field. Our benchmark includes two lightweight MLLMs, two downstream tasks, three evaluation metrics, and five datasets across three domains, along with six comparison baselines, covering over ten types of modality heterogeneities across four multimodal scenarios. To address the challenges posed by multimodal heterogeneity, we develop a general FedMLLM framework that integrates classic FL methods alongside two modality-agnostic strategies. Extensive experimental results show that our proposed FL paradigm improves the performance of MLLMs by broadening the range of training data and mitigating multimodal heterogeneity. Code is available in supplementary materials.