Text-driven Prompt Generation for Vision-Language Models in Federated Learning
This work addresses the problem of improving generalization in federated prompt learning for vision-language models, which is incremental as it builds on existing prompt learning techniques.
The paper tackled the challenge of prompt learning for vision-language models in federated learning, which struggles to generalize to unseen classes, by proposing Federated Text-driven Prompt Generation (FedTPG) that learns a context-aware prompt generation network across clients; results show it outperforms existing methods on nine datasets with better generalization to seen and unseen classes and unseen datasets.
Prompt learning for vision-language models, e.g., CoOp, has shown great success in adapting CLIP to different downstream tasks, making it a promising solution for federated learning due to computational reasons. Existing prompt learning techniques replace hand-crafted text prompts with learned vectors that offer improvements on seen classes, but struggle to generalize to unseen classes. Our work addresses this challenge by proposing Federated Text-driven Prompt Generation (FedTPG), which learns a unified prompt generation network across multiple remote clients in a scalable manner. The prompt generation network is conditioned on task-related text input, thus is context-aware, making it suitable to generalize for both seen and unseen classes. Our comprehensive empirical evaluations on nine diverse image classification datasets show that our method is superior to existing federated prompt learning methods, that achieve overall better generalization on both seen and unseen classes and is also generalizable to unseen datasets.