Federated Multimodal Learning with Dual Adapters and Selective Pruning for Communication and Computational Efficiency
This addresses the trade-off between model personalization and generalization for scalable real-world Federated Learning applications, though it is incremental.
The paper tackles the challenge of heterogeneous data distributions in Federated Learning, which leads to suboptimal global models, by proposing a dual-adapter framework with selective pruning. It achieves higher test accuracy, lower performance variance, improved worst-case performance, and reduces communication and computation costs in experiments on vision and language tasks.
Federated Learning (FL) enables collaborative learning across distributed clients while preserving data privacy. However, FL faces significant challenges when dealing with heterogeneous data distributions, which can lead to suboptimal global models that fail to generalize across diverse clients. In this work, we propose a novel framework designed to tackle these challenges by introducing a dual-adapter approach. The method utilizes a larger local adapter for client-specific personalization and a smaller global adapter to facilitate efficient knowledge sharing across clients. Additionally, we incorporate a pruning mechanism to reduce communication overhead by selectively removing less impactful parameters from the local adapter. Through extensive experiments on a range of vision and language tasks, our method demonstrates superior performance compared to existing approaches. It achieves higher test accuracy, lower performance variance among clients, and improved worst-case performance, all while significantly reducing communication and computation costs. Overall, the proposed method addresses the critical trade-off between model personalization and generalization, offering a scalable solution for real-world FL applications.