FedPower: Privacy-Preserving Distributed Eigenspace Estimation
This work addresses privacy and efficiency issues in distributed eigenspace estimation for organizations handling sensitive data, representing an incremental improvement over existing federated learning methods.
The paper tackles the challenge of eigenspace estimation in federated learning, where data privacy and communication efficiency are concerns, by proposing FedPower, a method that combines local power iterations with global aggregation and differential privacy, achieving improved communication efficiency and strong privacy protection with convergence bounds and experimental validation.
Eigenspace estimation is fundamental in machine learning and statistics, which has found applications in PCA, dimension reduction, and clustering, among others. The modern machine learning community usually assumes that data come from and belong to different organizations. The low communication power and the possible privacy breaches of data make the computation of eigenspace challenging. To address these challenges, we propose a class of algorithms called \textsf{FedPower} within the federated learning (FL) framework. \textsf{FedPower} leverages the well-known power method by alternating multiple local power iterations and a global aggregation step, thus improving communication efficiency. In the aggregation, we propose to weight each local eigenvector matrix with {\it Orthogonal Procrustes Transformation} (OPT) for better alignment. To ensure strong privacy protection, we add Gaussian noise in each iteration by adopting the notion of \emph{differential privacy} (DP). We provide convergence bounds for \textsf{FedPower} that are composed of different interpretable terms corresponding to the effects of Gaussian noise, parallelization, and random sampling of local machines. Additionally, we conduct experiments to demonstrate the effectiveness of our proposed algorithms.