Accelerating Differentially Private Federated Learning via Adaptive Extrapolation
This work addresses the efficiency and privacy trade-off in federated learning for resource-constrained clients, representing an incremental improvement over existing DP-FL methods.
The paper tackled the slow convergence problem in differentially private federated learning (DP-FL) due to client data heterogeneity, and the result was that their proposed DP-FedEXP method provably accelerates convergence and empirically outperforms existing methods without requiring additional hyperparameters or client computational cost.
The federated learning (FL) framework enables multiple clients to collaboratively train machine learning models without sharing their raw data, but it remains vulnerable to privacy attacks. One promising approach is to incorporate differential privacy (DP)-a formal notion of privacy-into the FL framework. DP-FedAvg is one of the most popular algorithms for DP-FL, but it is known to suffer from the slow convergence in the presence of heterogeneity among clients' data. Most of the existing methods to accelerate DP-FL require 1) additional hyperparameters or 2) additional computational cost for clients, which is not desirable since 1) hyperparameter tuning is computationally expensive and data-dependent choice of hyperparameters raises the risk of privacy leakage, and 2) clients are often resource-constrained. To address this issue, we propose DP-FedEXP, which adaptively selects the global step size based on the diversity of the local updates without requiring any additional hyperparameters or client computational cost. We show that DP-FedEXP provably accelerates the convergence of DP-FedAvg and it empirically outperforms existing methods tailored for DP-FL.