FedRA: A Random Allocation Strategy for Federated Tuning to Unleash the Power of Heterogeneous Clients
This addresses the problem of resource heterogeneity in federated tuning for researchers and practitioners, offering a practical solution that is incremental in adapting existing methods to handle client limitations.
The paper tackles the challenge of fine-tuning foundation models in federated learning with heterogeneous clients by proposing FedRA, a random allocation strategy that reorganizes model layers for resource-constrained clients, resulting in significant performance improvements on large-scale image datasets like DomainNet and NICO++ under non-iid settings.
With the increasing availability of Foundation Models, federated tuning has garnered attention in the field of federated learning, utilizing data and computation resources from multiple clients to collaboratively fine-tune foundation models. However, in real-world federated scenarios, there often exist a multitude of heterogeneous clients with varying computation and communication resources, rendering them incapable of supporting the entire model fine-tuning process. In response to this challenge, we propose a novel federated tuning algorithm, FedRA. The implementation of FedRA is straightforward and can be seamlessly integrated into any transformer-based model without the need for further modification to the original model. Specifically, in each communication round, FedRA randomly generates an allocation matrix. For resource-constrained clients, it reorganizes a small number of layers from the original model based on the allocation matrix and fine-tunes using adapters. Subsequently, the server aggregates the updated adapter parameters from the clients according to the current allocation matrix into the corresponding layers of the original model. It is worth noting that FedRA also supports scenarios where none of the clients can support the entire global model, which is an impressive advantage. We conduct experiments on two large-scale image datasets, DomainNet and NICO++, under various non-iid settings. The results demonstrate that FedRA outperforms the compared methods significantly. The source code is available at \url{https://github.com/leondada/FedRA}.