Inclusive Data Representation in Federated Learning: A Novel Approach Integrating Textual and Visual Prompt
This addresses data representation issues in Federated Learning for clients with multimodal data, but it appears incremental as it builds on existing prompt tuning methods.
The paper tackled the problem of communication overhead and data heterogeneity in Federated Learning by introducing Twin Prompt Federated Learning (TPFL) and Augmented TPFL (ATPFL), which integrate visual and textual prompts and use contrastive learning, resulting in superior performance compared to baselines.
Federated Learning (FL) is often impeded by communication overhead issues. Prompt tuning, as a potential solution, has been introduced to only adjust a few trainable parameters rather than the whole model. However, current single-modality prompt tuning approaches fail to comprehensively portray local clients' data. To overcome this limitation, we present Twin Prompt Federated learning (TPFL), a pioneering solution that integrates both visual and textual modalities, ensuring a more holistic representation of local clients' data characteristics. Furthermore, in order to tackle the data heterogeneity issues, we introduce the Augmented TPFL (ATPFL) employing the contrastive learning to TPFL, which not only enhances the global knowledge acquisition of client models but also fosters the development of robust, compact models. The effectiveness of TPFL and ATPFL is substantiated by our extensive evaluations, consistently showing superior performance compared to all baselines.