Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method
This work provides theoretical insights for researchers and practitioners in federated learning, but it is incremental as it builds on existing prompt-based methods without introducing a new paradigm.
The paper tackles the lack of theoretical analysis for prompt-based federated learning with vision-language models by developing a framework using feature learning theory, showing that performance depends on the ratio of task-relevant to task-irrelevant coefficients and introducing a prompt portfolio method that balances generalization and personalization with an optimal mixing coefficient.
Integrating pretrained vision-language foundation models like CLIP into federated learning has attracted significant attention for enhancing generalization across diverse tasks. Typically, federated learning of vision-language models employs prompt learning to reduce communication and computational costs, i.e., prompt-based federated learning. However, there is limited theoretical analysis to understand the performance of prompt-based federated learning. In this work, we construct a theoretical analysis framework for prompt-based federated learning via feature learning theory. Specifically, we monitor the evolution of signal learning and noise memorization in prompt-based federated learning, demonstrating that performance can be assessed by the ratio of task-relevant to task-irrelevant coefficients. Furthermore, we draw an analogy between income and risk in portfolio optimization and the task-relevant and task-irrelevant terms in feature learning. Leveraging inspiration from portfolio optimization that combining two independent assets will maintain the income while reducing the risk, we introduce two prompts: global prompt and local prompt to construct a prompt portfolio to balance the generalization and personalization. Consequently, we showed the performance advantage of the prompt portfolio and derived the optimal mixing coefficient. These theoretical claims have been further supported by empirical experiments.