Towards Sparsified Federated Neuroimaging Models via Weight Pruning
This addresses communication and privacy challenges in federated learning for neuroimaging, offering incremental improvements by combining existing pruning techniques with federated training.
The paper tackles the high communication costs in federated training of large neural networks by proposing FedSparsify, which prunes model weights during training, achieving up to 95% sparsity without performance loss on brain age prediction tasks, even with heterogeneous data, and also improves privacy by reducing susceptibility to membership inference attacks.
Federated training of large deep neural networks can often be restrictive due to the increasing costs of communicating the updates with increasing model sizes. Various model pruning techniques have been designed in centralized settings to reduce inference times. Combining centralized pruning techniques with federated training seems intuitive for reducing communication costs -- by pruning the model parameters right before the communication step. Moreover, such a progressive model pruning approach during training can also reduce training times/costs. To this end, we propose FedSparsify, which performs model pruning during federated training. In our experiments in centralized and federated settings on the brain age prediction task (estimating a person's age from their brain MRI), we demonstrate that models can be pruned up to 95% sparsity without affecting performance even in challenging federated learning environments with highly heterogeneous data distributions. One surprising benefit of model pruning is improved model privacy. We demonstrate that models with high sparsity are less susceptible to membership inference attacks, a type of privacy attack.