Automatic Structured Pruning for Efficient Architecture in Federated Learning
This addresses efficiency problems for federated learning systems with limited computational resources, though it is incremental as it builds on existing pruning methods.
The paper tackles the challenge of efficient model training on resource-constrained client devices in Federated Learning by proposing an automatic structured pruning scheme, which reduces parameters by 89% and FLOPS by 90% while maintaining accuracy and cutting communication overhead by up to 5x.
In Federated Learning (FL), training is conducted on client devices, typically with limited computational resources and storage capacity. To address these constraints, we propose an automatic pruning scheme tailored for FL systems. Our solution improves computation efficiency on client devices, while minimizing communication costs. One of the challenges of tuning pruning hyper-parameters in FL systems is the restricted access to local data. Thus, we introduce an automatic pruning paradigm that dynamically determines pruning boundaries. Additionally, we utilized a structured pruning algorithm optimized for mobile devices that lack hardware support for sparse computations. Experimental results demonstrate the effectiveness of our approach, achieving accuracy comparable to existing methods. Our method notably reduces the number of parameters by 89% and FLOPS by 90%, with minimal impact on the accuracy of the FEMNIST and CelebFaces datasets. Furthermore, our pruning method decreases communication overhead by up to 5x and halves inference time when deployed on Android devices.