Super-convergence and Differential Privacy: Training faster with better privacy guarantees
This addresses the efficiency and utility challenges for practitioners using differential privacy in machine learning, though it is incremental as it adapts an existing technique to a specific domain.
The paper tackles the problem of high training time and resource use in differentially private neural networks by applying super-convergence, resulting in an order-of-magnitude speedup and higher validation accuracies compared to baseline models.
The combination of deep neural networks and Differential Privacy has been of increasing interest in recent years, as it offers important data protection guarantees to the individuals of the training datasets used. However, using Differential Privacy in the training of neural networks comes with a set of shortcomings, like a decrease in validation accuracy and a significant increase in the use of resources and time in training. In this paper, we examine super-convergence as a way of greatly increasing training speed of differentially private neural networks, addressing the shortcoming of high training time and resource use. Super-convergence allows for acceleration in network training using very high learning rates, and has been shown to achieve models with high utility in orders of magnitude less training iterations than conventional ways. Experiments in this paper show that this order-of-magnitude speedup can also be seen when combining it with Differential Privacy, allowing for higher validation accuracies in much fewer training iterations compared to non-private, non-super convergent baseline models. Furthermore, super-convergence is shown to improve the privacy guarantees of private models.