DMM: Distributed Matrix Mechanism for Differentially-Private Federated Learning Based on Constant-Overhead Linear Secret Resharing
This work addresses the problem of enhancing privacy and utility in federated learning for applications requiring secure data handling, representing an incremental improvement over existing distributed DP mechanisms.
The paper tackled the trade-off between privacy and utility in federated learning by introducing a distributed matrix mechanism that combines the better privacy of distributed differential privacy with the improved utility of the matrix mechanism, achieving significant utility gains with minimal overhead.
Federated Learning (FL) solutions with central Differential Privacy (DP) have seen large improvements in their utility in recent years arising from the matrix mechanism, while FL solutions with distributed (more private) DP have lagged behind. In this work, we introduce the distributed matrix mechanism to achieve the best-of-both-worlds; better privacy of distributed DP and better utility from the matrix mechanism. We accomplish this using a novel cryptographic protocol that securely transfers sensitive values across client committees of different training iterations with constant communication overhead. This protocol accommodates the dynamic participation of users required by FL, including those that may drop out from the computation. We provide experiments which show that our mechanism indeed significantly improves the utility of FL models compared to previous distributed DP mechanisms, with little added overhead.