Distributed quasi-Newton robust estimation under differential privacy
This work addresses privacy and robustness in distributed computing for applications like federated learning, though it appears incremental as it builds on existing quasi-Newton and privacy techniques.
The paper tackles the problem of distributed estimation with Byzantine machines under privacy constraints by developing a robust quasi-Newton method that reduces transmission costs and privacy budgeting, achieving optimal convergence rates and asymptotic normality.
For distributed computing with Byzantine machines under Privacy Protection (PP) constraints, this paper develops a robust PP distributed quasi-Newton estimation, which only requires the node machines to transmit five vectors to the central processor with high asymptotic relative efficiency. Compared with the gradient descent strategy which requires more rounds of transmission and the Newton iteration strategy which requires the entire Hessian matrix to be transmitted, the novel quasi-Newton iteration has advantages in reducing privacy budgeting and transmission cost. Moreover, our PP algorithm does not depend on the boundedness of gradients and second-order derivatives. When gradients and second-order derivatives follow sub-exponential distributions, we offer a mechanism that can ensure PP with a sufficiently high probability. Furthermore, this novel estimator can achieve the optimal convergence rate and the asymptotic normality. The numerical studies on synthetic and real data sets evaluate the performance of the proposed algorithm.