Privacy-Preserving Personalized Federated Prompt Learning for Multimodal Large Language Models
This work addresses privacy-preserving personalization in federated learning for multimodal AI systems, which is an incremental improvement on existing methods.
The paper tackles the challenge of balancing personalization, generalization, and privacy in federated prompt learning for multimodal LLMs by proposing a differentially private federated prompt learning approach that uses low-rank factorization and applies local and global differential privacy to prompts, with experiments showing effectiveness over benchmarks.
Multimodal Large Language Models (LLMs) are pivotal in revolutionizing customer support and operations by integrating multiple modalities such as text, images, and audio. Federated Prompt Learning (FPL) is a recently proposed approach that combines pre-trained multimodal LLMs such as vision-language models with federated learning to create personalized, privacy-preserving AI systems. However, balancing the competing goals of personalization, generalization, and privacy remains a significant challenge. Over-personalization can lead to overfitting, reducing generalizability, while stringent privacy measures, such as differential privacy, can hinder both personalization and generalization. In this paper, we propose a Differentially Private Federated Prompt Learning (DP-FPL) approach to tackle this challenge by leveraging a low-rank factorization scheme to capture generalization while maintaining a residual term that preserves expressiveness for personalization. To ensure privacy, we introduce a novel method where we apply local differential privacy to the two low-rank components of the local prompt, and global differential privacy to the global prompt. Our approach mitigates the impact of privacy noise on the model performance while balancing the tradeoff between personalization and generalization. Extensive experiments demonstrate the effectiveness of our approach over other benchmarks.