Statistical Inference for Differentially Private Stochastic Gradient Descent
This work addresses the need for reliable statistical inference in privacy-preserving machine learning for sensitive data analysis, representing an incremental advancement by adapting existing methods to the specific requirements of DP-SGD.
The paper tackled the problem of statistical inference for Differentially Private Stochastic Gradient Descent (DP-SGD) by establishing asymptotic properties under randomized subsampling and extending them to DP-SGD, showing that the asymptotic variance decomposes into statistical, sampling, and privacy-induced components, and proposing two methods for constructing valid confidence intervals that achieve nominal coverage rates while maintaining privacy.
Privacy preservation in machine learning, particularly through Differentially Private Stochastic Gradient Descent (DP-SGD), is critical for sensitive data analysis. However, existing statistical inference methods for SGD predominantly focus on cyclic subsampling, while DP-SGD requires randomized subsampling. This paper first bridges this gap by establishing the asymptotic properties of SGD under the randomized rule and extending these results to DP-SGD. For the output of DP-SGD, we show that the asymptotic variance decomposes into statistical, sampling, and privacy-induced components. Two methods are proposed for constructing valid confidence intervals: the plug-in method and the random scaling method. We also perform extensive numerical analysis, which shows that the proposed confidence intervals achieve nominal coverage rates while maintaining privacy.