Timothy Stevens

Timothy Stevens, Ivoline C. Ngong, David Darais et al.

We present backpropagation clipping, a novel variant of differentially private stochastic gradient descent (DP-SGD) for privacy-preserving deep learning. Our approach clips each trainable layer's inputs (during the forward pass) and its upstream gradients (during the backward pass) to ensure bounded global sensitivity for the layer's gradient; this combination replaces the gradient clipping step in existing DP-SGD variants. Our approach is simple to implement in existing deep learning frameworks. The results of our empirical evaluation demonstrate that backpropagation clipping provides higher accuracy at lower values for the privacy parameter $ε$ compared to previous work. We achieve 98.7% accuracy for MNIST with $ε= 0.07$ and 74% accuracy for CIFAR-10 with $ε= 3.64$.

Timothy Stevens, Joseph Near, Christian Skalka

Secure Multiparty Computation (MPC) can improve the security and privacy of data owners while allowing analysts to perform high quality analytics. Secure aggregation is a secure distributed mechanism to support federated deep learning without the need for trusted third parties. In this paper we present a highly performant secure aggregation protocol with sub-linear communication complexity. Our protocol achieves greater communication and computation efficiencies through a group-based approach. It is similar to secret sharing protocols extended to vectors of values-aka gradients-but within groups we add an additional layer of secret sharing of shares themselves-aka sharding. This ensures privacy of secret inputs in the standard real/ideal security paradigm, in both semi-honest and malicious settings where the server may collude with the adversary. In the malicious setting with 5% corrupt clients and 5% dropouts, our protocol can aggregate over a federation with 100,000,000 members and vectors of length 100 while requiring each client to communicate with only 350 other clients. The concrete computation cost for this aggregation is less than half a second for the server and less than 100ms for the client.