Federated Dropout -- A Simple Approach for Enabling Federated Learning on Resource Constrained Devices
This is an incremental improvement for federated learning systems on resource-constrained devices.
The paper tackles the challenge of resource-constrained devices in federated learning by proposing Federated Dropout (FedDrop), which uses heterogeneous dropout rates to generate subnets, reducing communication and computation loads while outperforming conventional FL in overfitting cases and uniform dropout schemes.
Federated learning (FL) is a popular framework for training an AI model using distributed mobile data in a wireless network. It features data parallelism by distributing the learning task to multiple edge devices while attempting to preserve their local-data privacy. One main challenge confronting practical FL is that resource constrained devices struggle with the computation intensive task of updating of a deep-neural network model. To tackle the challenge, in this paper, a federated dropout (FedDrop) scheme is proposed building on the classic dropout scheme for random model pruning. Specifically, in each iteration of the FL algorithm, several subnets are independently generated from the global model at the server using dropout but with heterogeneous dropout rates (i.e., parameter-pruning probabilities),each of which is adapted to the state of an assigned channel. The subnets are downloaded to associated devices for updating. Thereby, FedDrop reduces both the communication overhead and devices' computation loads compared with the conventional FL while outperforming the latter in the case of overfitting and also the FL scheme with uniform dropout (i.e., identical subnets).