Amplifying Rényi Differential Privacy via Shuffling
This work addresses privacy concerns in large-scale machine learning implementations, providing theoretical support for a faster training algorithm, though it is incremental as it extends existing privacy analysis to a specific variant.
The paper tackles the problem of analyzing the privacy guarantees of cyclic stochastic gradient descent (SGD) under Rényi differential privacy, showing that its privacy bounds are competitive with those of traditional SGD with replacement sampling.
Differential privacy is a useful tool to build machine learning models which do not release too much information about the training data. We study the Rényi differential privacy of stochastic gradient descent when each training example is sampled without replacement (also known as cyclic SGD). Cyclic SGD is typically faster than traditional SGD and is the algorithm of choice in large-scale implementations. We recover privacy guarantees for cyclic SGD which are competitive with those known for sampling with replacement. Our proof techniques make no assumptions on the model or on the data and are hence widely applicable.